Learning from failures: First Experiment at NectarCommerce Extension Approach

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

Learning from failures: First Experiment at NectarCommerce Extension Approach

Where we left off

In the past few posts we have learned how to write code that extends existing models, routers, added methods to override view rendering and run multiple phoenix application together in the same umbrella project. Let’s Continue to build upon that and write our first extension for NectarCommerce favorite products and ultimately our store based on NectarCommerce.

A layered guide to NectarCommerce extensions

Index

  1. Setup
  2. Model Layer
  3. View Layer
  4. Testing

Setup

Create a new phoenix application to hold the favorite products application, in your shell run inside the umbrella/apps folder:

We could have gone with a regular mix application, but Phoenix/Ecto will be handy in this case, since we want to have views to display stuff and a model to store data.

While we are at it let’s configure our dev.exs and test.exs to use the same db as Nectar, we could write some code and share the db settings between Nectar and our extensions, see: running multiple elixir applications together for more details. But, for simplicity’s sake we are just copying the settings from Nectar to get started.

DB_SETTINGS:

We need to let the extension manager know that this application is an extension for Nectar.
Update the dependencies in extension_manager/mix.exs with the favorite_products dependency.

That should be enough to get us going.

MODEL LAYER

We want a Nectar user to have some products to like and a way to remember them, we can use a join table with relations to user and products to achieve this. let’s generate the model:

Now to point to correct Nectar models. Open up the source and change the associations from favorite products model to Nectar models. In the end we have a schema like:

Of, course this is only the extension view of this relationship, We want the Nectar user to be aware of this relationship and most important of all, we should be able to do something like Nectar.User.liked_products(user) to fetch the products liked by user.

Time to call our handy macros written earlier to perform the compile time code injection. Let’s create the Nectar_extension.ex file in favorite_products/lib/ directory and place this code there:

Don’t forget to update the install file in extensions_manager.

Now we have a user that can like products and product from which we can query which users liked it.

Time to play with what we have built so far, start a shell in Nectar folder with:


Oops!, forgot the migration, remember we shared the db config earlier let’s put that to use and run:

Which will migrate the user_likes table onto the original Nectar database. Back to our shell

Voila! we can now save and retrieve records to a relation we defined outside Nectar from Nectar models without actually modifying Nectar code.

VIEW LAYER

Now that we can save the user likes, we should probably add an interface for the user to like them as well. Which leads us to the first shortcoming, in our current approach, we can replace existing views but right now we don’t have anything for adding to an existing view(Please leave us a note here if you know of a clean & performant method to do this). I also suspect most of us will end up overriding the existing views to something more custom then updating it piecemeal via extensions. For now let’s have a page where we list all the products and user can mark them as liked or unlike the previously liked ones.

controller

Notice how we use the Nectar.Repo itself instead of using the FavoriteProducts.Repo.
In-fact besides migration, we won’t be utilizing or starting the FavoriteProducts.Repo, which will help us keep the number of connections open to database limited only to the Nectar.Repo.

the view file: index.html.eex

In both of the files we refer to routes via NectarRoutes alias instead of favorite products.
To add the route from Nectar, update nectar_extension.ex with the following code:

And add to install.ex the calls:

Now we can see the added routes from Nectar:

So far so good, we have modified and added routes and controller to Nectar’s router. Time to see our handiwork in action, start the server from Nectar application with:

And, visit 127.0.0.1:4000/favorite and click on mark to like a product.

Missing Layout

But things don’t seem right do they? Our Nectar layout has been replaced with the default one used by phoenix. Let’s rectify that.

Update layout_view.ex as:

and recompile and restart the server

On our next visit:

Layout Present

Much better.

Note: When we need to change the extension code while running the server we will have to recompile and reload. We don’t have anything in Nectar right now to monitor all extensions file and do an automatic compilation and code reload.

Testing

We are almost done now. To ensure that we know when things break we should consider adding a few test cases. For that we need to make sure that Nectar migrations are run before running the migrations for favorite products and we need the Nectar repo running as well.

For the former we could update the test_helper.ex with:

But things are not so smooth for accessing the Nectar Repo. Which brings us to what we think is the ultimate downfall of this approach:

An Untestable soution

Ideally, running mix test should work and we should see our test running green, unfortunately this requires Nectar to be compiled before running the tests, which is impossible since Nectar depends upon the extension_manager to be compiled which depends upon all the extensions to be compiled, resulting in a cyclic dependency. Also we used Nectar’s repo for all database work. That works because we were running our server through Nectar and the repo was started in Nectar’s Supervision tree. Which again adds an implicit requirement i.e. Nectar application is available and ready to be started during test time. We could replace Nectar.Repo with FavoriteProducts.Repo using compile time conditionals i.e. when MIX_ENV==test, but it is a can of worms we would rather avoid right now.

This seems like the end of the road for this approach. Where we are failing right now is making Nectar available to extensions as a dependency at compile time and in turn test time. So that they can run independently. Let’s try that in our second approach and reverse the dependency order.

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Extension Approach Explained

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

What will be NectarCommerce

Off-the-shelf Opensource E-commerce application for building an online store.

Provides an Extension Framework to support features not included in core as extensions.

Strives for unobtrusive parallel development of NectarCommerce and Extensions

NectarCommerce is committed to providing a ready-to-use e-commerce solution but the definition of 100% is different under different business domains. It aims to solve common use-cases as part of the project and relying on extension framework to tap the rest.

NectarCommerce Extension Approach

This post aims at documenting how the approach we took to make NectarCommerce extensible is structured, the order of dependencies amongst the various nuts and bolts of NectarCommerce and its extensions as implemented.

NectarCommerce can be divided in two components, both of which reside in the same umbrella application.

  1. Nectar: The phoenix project, which acts as the backbone of your E-Commerce application.
  2. Extension Manager: A plugin manager/DSL provider for Nectar. It also owns the compile time step for Nectar. We intend to make the DSL/Compiler as a separate package and make Extension Manager solely responsible for downloading and installing extensions.

Application Overview

Nectar

Nectar is a run of the mill phoenix application, with models, controllers and some conveniences built into it which makes it amenable to extension. It has no dependencies on other application in the umbrella besides worldly which once done should also be a separate package.

  1. Nectar.Extender, if any module is using this during compile time, Nectar.Extender searches for whether the module which can act as an extension to the using module has been compiled and loads the extension into it. There is a basic naming convention being followed for doing this: if Nectar.Product uses Nectar.Extender then it will search for module ExtensionsManager.ExtendProduct as the module to use for extension. This is a proof of concept implementation, which we can flesh out later to support fully name-spaced extension modules.
    source.

    Note: This currently limits the point of extension per module to a single module. It is by design to keep things in one place.

  2. Nectar.RouteExtender, Similar to Nectar.Extender except it is specialized for extending routes.

So any extension that needs to modify Nectar, needs to utilize the DSL provided by extension manager, we will go through this in detail when we expand on how to build an extension.

Extension Manager

This is a regular mix application that will be used to fetch the extensions and provides a thin DSL(for more details on this see our posts on Model Extension, View Extension and Router Extension) which can be used to express how changes are to made to Nectar. Also any extension that we need to install will be added here as a dependency and properly hooked into the correct modules (Please see the example implementation for one way of doing this). This module provides 3 components for building and compiling Nectar extensions.

  1. DSL : A simple DSL for declaring components that need to be injected into models/router. See our previous posts for how the DSL looks and behaves.

  2. Extension Compiler: It is a basic compile time step that marks files which are using Nectar.Extender(i.e. use Nectar.Extender) for recompilation so that they pick up any changes made to the extensions. It is currently based on how phoenix compiler works.

  3. Install: Extensions can document how they are to be installed by declaring the code using DSL inside method calls and describing how and which modules to call these methods in. These instructions are then followed to compile the injection payload. If this seems cryptic/vague, please refer to the example implementation of favorite products extensions on how the install file is structured.

Dependencies and Code loading

Extensions Manager & Nectar : Extension Manager does not depend upon Nectar directly(it may be a transitive dependency via the extensions) neither does Nectar depend upon Extension Manager. Nectar searches for modules in the umbrella via Code.ensure_loaded to find if extensions exist. While it’s not ideal and as explicit as we want it to be, it is a pragmatic solution for what it allows us to achieve which is basically a form of mutual dependency.

Extensions & Nectar: Extensions should depend upon nectar. Again this may seem counterintuitive since Nectar will be enhanced via extensions, but ultimately we will need the Nectar dependency for running tests, pattern matching on Nectar structs and models and for building exrm releases(more on this later). After we are done we can always recompile nectar to use the extensions.

This concludes our high level description of how the different parts of NectarCommerce interact with each other. Lets continue and see how we can utilize the above infrastructure for building extensions in our next post.

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Phoenix Router Extension

The post belongs to NectarCommerce and Extension Framework Awareness

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

What will be NectarCommerce

Off-the-shelf Opensource E-commerce application for building an online store.

Provides an Extension Framework to support features not included in core as extensions.

Strives for unobtrusive parallel development of NectarCommerce and Extensions

NectarCommerce is committed to providing a ready-to-use e-commerce solution but the definition of 100% is different under different business domains.It aims to solve common use-cases as part of the project and relying on extension framework to tap the rest.

Phoenix Router Extension

Note: This blog post is very similar to Ecto Model Schema Extension, so if you are already familiar with it, you can jump straight to the final version

Why

We want to allow Extensions to add routes into Nectar Router without modifying the Nectar Router source.

How

Minimum three parts are needed to create & use an extension effectively:

  • Library Code
  • Service Code
  • Consumer Code

An extension and its use with Nectar can be viewed as Producer / Consumer relationship bound by a communication protocol.

Extension which wants to add a route, say a list of favorites, is a Producer (Service Code).

Nectar Router is a Consumer (Consumer Code) allowing the route additions through a communication protocol (Library Code)

Let’s begin the journey of incremental changes to bring consumer, service and library code into existence starting from a simple use-case of adding a route for showing favorites.

Note: Please refer Introduction to Metaprogramming for more information on Metaprogramming in Elixir

  1. A straightforward way to add favorites route in Nectar.Router would be adding it directly in Nectar.Router, see full version here, but it requires change in Nectar source. Let’s move to next step for avoiding any modification to Nectar.Router

  2. We can add a function to Nectar to which other extensions can delegate the responsibility of route additions, see full version here. See Nectar.ExtendRouter example below on how to use it

  3. Now, with delegation function mount in place, we can work towards providing a way to register the routes to be added, see full version here. Please check the usage of Module attributes for same below.

  4. Earlier, Module.put_attribute need to be used multiple times to define multiple routes whereas now we wrapped it in an anonymous function to encapsulate the collection of routes through a simple and consistent interface, see full version here. There can be multiple extensions used for different functionality and hence multiple routes need to be registered and defined

  5. Now, Nectar.ExtendRouter is getting cluttered with ancillary method definitions. Let’s move it out to another module and use it, see full version here

  6. Let’s further reduce the boilerplate of registering defined_routes module attribute and importing define_route method definition with using callback, see full version here

  7. Reference of defined_routes Module attribute is scattered across Nectar.RouterExtender and Nectar.RouterExtension so let’s move it out to Nectar.RouterExtension to consolidate the usage via __before_compile__ and definition together, see full version here

  8. With above changes, it’s now possible to define routes any number of times, see full version here. Also, routes can now be added using define_route in Nectar.ExtendRouter without making any changes to Nectar.Router.

Check all the revisions at once, here

 Now, in the final version, you can easily find the three components, consumer, service and library code, as desired in extensible system

Please refer the demonstration approach for library code, service code and consumer code as used with favorite products extension

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Ecto Model Support Functions Extension

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

What will be NectarCommerce

Off-the-shelf Opensource E-commerce application for building an online store.

Provides an Extension Framework to support features not included in core as extensions.

Strives for unobtrusive parallel development of NectarCommerce and Extensions

NectarCommerce is committed to providing a ready-to-use e-commerce solution but the definition of 100% is different under different business domains. It aims to solve common use-cases as part of the project and relying on extension framework to tap the rest.

Ecto Model Support Functions Extension

Note: This blog post is very similar to Ecto Model Schema Extension, so if you are already familiar with it, you can jump straight to the final version

Why

We want to allow Extensions to add support functions to existing Nectar Models without changing the Nectar Models.

How

Minimum three parts are needed to create & use an extension effectively:

  • Library Code
  • Service Code
  • Consumer Code

An extension and its use with Nectar can be viewed as Producer / Consumer relationship bound by a communication protocol.

Extension which wants to add a function, say fn_from_outside to Nectar Product Model, is a Producer (Service Code).

Nectar Model is a Consumer (Consumer Code) allowing the new function additions through a communication protocol (Library Code)

Let’s begin the journey of incremental changes to bring consumer, service and library code into existence starting from a simple use-case of adding a function, say fn_from_outside, to Nectar Product.

Note: Please refer Introduction to Metaprogramming for more information on Metaprogramming in Elixir

  1. A straightforward way to add a function, say fn_from_outside, to Nectar Product would be adding it directly in Nectar.Product, but it requires change in Nectar source. Let’s move to the next step for avoiding any modification to Nectar.Product

  2. Like, other extensions, adding a function to Nectar Model and importing it would not solve our purpose of calling function as Nectar.Product.fn_from_outside.
    import <Module> makes the <Module> functions available inside the module and can only be called inside Module and not from outside like Nectar.Product.fn_from_outside

    As expected, Elixir is well-aware of the use-case and provides @before_compile <Module> hook to inject functions in Modules as if their own and can be called as Nectar.Product.fn_from_outside, see full version here. See Nectar.ExtendProduct example below on how to use it.


  3. Now, with @before_compile Nectar.ExtendProduct in place, we can work towards providing a way to register the new functions, see full version here. Please check the usage of Module attributes for same below.

  4. Earlier, Module.put_attribute was used multiple times to define many functions whereas now we wrapped it in an anonymous function to encapsulate the collection of function additions through a simple and consistent interface, see full version here. There can be multiple extensions used for different functionality and hence many functions need to be registered and defined

  5. Now, Nectar.ExtendProduct is getting cluttered with ancillary method definitions. Let’s move it out to another module and use it, see full version here

  6. Let’s further reduce the boilerplate of registering method_block module attribute and importing include_method method definition with __using__ callback, see full version here


  7. Reference of method_block Module attribute is scattered across Nectar.ExtendProduct and Nectar.ModelExtension so let’s move it out to Nectar.ModelExtension to consolidate the usage via __before_compile__ and definition together, see full version here

  8. Now, Nectar.ExtendProduct has __using__ macro definition, which can also be moved to Nectar.ModelExtension to just have new method defintions in Nectar.ExtendProduct, see full version here

  9. With above changes, it’s now possible to define any number of functions, see full version here. Also, any number of functions can now be added using include_method in Nectar.ExtendProduct without making any changes to Nectar.Product.

Check all the revisions at once, here

  Now, in the final version, you can easily find the three components, consumer, service and library code, as desired in extensible system

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Ecto Model Schema Extension

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

What will be NectarCommerce

Off-the-shelf Opensource E-commerce application for building an online store.

Provides an Extension Framework to support features not included in core as extensions.

Strives for unobtrusive parallel development of NectarCommerce and Extensions

NectarCommerce is committed to providing a ready-to-use e-commerce solution but the definition of 100% is different under different business domains. It aims to solve common use-cases as part of the project and relying on extension framework to tap the rest.

Ecto Model Schema Extension

Why

We want to allow Extensions to modify the schema of existing Nectar Models without changing the Nectar Models.

Extensions should be able to add new fields and associations to existing models as and when needed.

How

Minimum three parts are needed to create & use an extension effectively:

  • Library Code
  • Service Code
  • Consumer Code

An extension and its use with Nectar can be viewed as Producer / Consumer relationship bound by a communication protocol.

Extension which wants to add a virtual field, say special, to Nectar Product Model Schema is a Producer (Service Code).

Nectar Model is a Consumer (Consumer Code) allowing the schema changes through a communication protocol (Library Code)

Let’s begin the journey of incremental changes to bring consumer, service and library code into existence starting from a simple use-case of adding a virtual boolean field, say special, to Nectar Product.

Note: Please refer Introduction to Metaprogramming for more information on Metaprogramming in Elixir

  1. A straightforward way to add a virtual field, say special, to Nectar Product would be adding it directly in Nectar.Product Schema definition, but it requires change in Nectar source. Let’s move to the next step for avoiding any modification to Nectar.Product

  2. We can add a function to Nectar Model Schema to which other extensions can delegate the responsibility of schema changes, see full version here. See Nectar.ExtendProduct example below on how to use it.

  3. Now, with delegation function extensions in place, we can work towards providing a way to register the schema changes, see full version here. Please check the usage of Module attributes for same below.

  4. Earlier, Module.put_attribute was used multiple times to define multiple schema changes whereas now we wrapped it in an anonymous function to encapsulate the collection of schema changes through a simple and consistent interface, see full version here. There can be multiple extensions used for different functionality and hence multiple schema changes need to be registered and defined

  5. Now, Nectar.ExtendProduct is getting cluttered with ancillary method definitions. Let’s move it out to another module and use it, see full version here

  6. Let’s further reduce the boilerplate of registering schema_changes module attribute and importing add_to_schema method definition with using callback, see full version here

  7. Reference of schema_changes Module attribute is scattered across Nectar.ExtendProduct and Nectar.ModelExtension so let’s move it out to Nectar.ModelExtension to consolidate the usage via __before_compile__ and definition together, see full version here

  8. With above changes, it’s now possible to define schema changes any number of times, see full version here. Also, schema changes can now be added using include_method in Nectar.ExtendProduct without making any changes to Nectar.Product.

Check all the revisions at once, here

Now, in the final version, you can easily find the three components, consumer, service and library code, as desired in extensible system

Please refer the demonstration approach for library code, service code and consumer code as used with favorite products extension

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Privacy Preference Center