Building an exrm release including NectarCommerce

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

Building an exrm release including NectarCommerce

You can read more about how to build an exrm release for an umbrella project here. Following the example outlined in the repository, the main app for our case will be the User Store Application. Let’s get started and try to build an exrm release:

Note: We are going to build our release in Dev environment, the only thing that should change for production is the configuration. Some of the options may already be switched off for the production release. See Phoenix Guides for how to build production exrm release for phoenix applications.

Step 1:
Update deps of umbrella to include exrm and run mix deps.get to install them.

Step 2:
Run the mix release command. Let’s try it with the dev environment for now.

Let’s examine the output and keep on continuing:

In the end we can see:

Let’s try accessing it. Running with no args gives us the output:

We need to start the application to access the user_app and nectar, so let’s do that. We are using foreground to see the console output, it can also be started to run as a background service. Doing that:

and we see

It seems to be unable to start the endpoint because the code reloader is missing. We can add it to the applications, but since it has no reason for belonging in the release, we will comment out the config for now in dev.exs and set code_reload: false in config.exs, doing that and building a new release.

It is working. Now to try accessing the server, it fails with connection refused, the server is not starting automatically.

Going over the phoenix endpoint configuration, we need to set server: true in endpoint configuration which is done automatically when running mix phoenix.server, doing that and rebuilding the release.

It’s alive! But we are getting 500 error. Hopping over to the logs(remember we are running it in the foreground mode)

It fails because guardian is not loaded. This was one of the missing dependencies which were not included in running applications. There were four of them:

  1. guardian
  2. arc
  3. comeonin
  4. phoenix_live_reload

Since we have already tackled the last one, let’s add the first three in the application list in nectar.

And trying again:

Success. Similarly, we can copy over the config for production environment when building our exrm release. Now we can deploy the user app as the main application, See phoenix guides for details on how to configure and build exrm releases for production.

Running Migrations for an Exrm Release

Please refer the excellent post for how to run migrations for an exrm release by @plataformatec.

You can always run mix release.clean -- implode to clean up your workspace.

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Developing NectarCommerce Extensions

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

Developing NectarCommerce Extensions

Where we left off

In the past few posts we have learned how to write code that extends existing models, routers, added methods to override view rendering and run multiple phoenix application together in the same umbrella project. Let’s Continue to build upon that and write our first extension for NectarCommerce favorite products and ultimately our store based on NectarCommerce.

For our curious readers, we tried another extension approach which had linear dependency (nectar => extensions_manager => extensions) helpful in development but turned out to had the serious limitation of Nectar being unavailable for testing.

To overcome the barrier of testing and pleasant developer workflow, what we need is that Nectar is available and compiled while developing extensions and if an extension is added it should recompile itself to include the new extensions.
We have modified Nectar for this approach to seek for extensions and used a custom compiler step(A story for another time) to mark files for recompilation. Let’s get started and see if we can scale the testing barrier.

A layered guide to NectarCommerce extensions

Index

  1. Setup
  2. Model Layer
  3. View Layer
  4. Starting the server
  5. Testing
  6. Creating your store with NectarCommerce
  7. Suggested Workflow

Setup

Create a new phoenix application to hold the favorite products application.
In your shell run inside the umbrella/apps folder:

We could have gone with a regular mix application, but phoenix/ecto will come in handy in this case, since we want to have views to display stuff and a model to store data.

While we are at it let’s configure our dev.exs to use the same db as Nectar, we could write some code and share the db settings between Nectar and our extensions see: running multiple phoenix application together for more details. But now for simplicity’s sake we are just copying the settings from Nectar to get started.

DB_SETTINGS:

We need to let the extension manager know that this application is an extension for Nectar.
Update the dependencies in extension_manager/mix.exs with the favorite_products dependency.

And now for the big differentiator, we will add Nectar as dependency of the favorite_products extension, effectively ensuring it is compiled before the extension.

Model Layer

We want a NectarCommerce user to have some products to like and a way to remember them in short a join table and with two associations let’s generate them:

Now to point to correct Nectar models. Open up the source and change the associations to from favorite products model to Nectar models. In the end we have a schema like:

Fun Fact: Since we are depending upon Nectar now we can use Nectar.Web, :model instead of FavoriteProducts.Web, :model in user_like.ex and make our extensions available for extension.

Of, course this is only the extension view of this relationship, We want the Nectar user to be aware of this relationship and most important of all, we should be able to do something like Nectar.User.liked_products(user) to fetch the liked products of the user.

Calling our handy macros to perform the dark art of compile time code injection. Let’s create the nectar_extension.ex file in favorite_products/lib/ directory and place this code there:

Don’t forget to update the install file in extensions_manager.

Now we have a user that can like products and product from which we can query what users liked it. Time to play.

From the root of umbrella, run the following to start the shell:

This should trigger another round of compilation. Ultimately loading the extension code into Nectar. Lets see if we were successful. But before doing that we should migrate the database.

Voila, we can now save and retrieve records to a relation we defined outside Nectar from Nectar models.

VIEW LAYER

Now that we can save the user likes, we should probably add an interface for the user to like them as well. Which leads us to the first shortcoming in our current approach, we can replace existing views but right now we don’t have anything for adding to an existing view(Please leave us a note here if you know of a clean performant approach to do this). Meanwhile we expect most people will end up overriding the existing views to something more custom then updating it piecemeal but i digress. For now let’s have a page where we list all the products and user can mark them as liked or unlike the previously liked ones.

controller

Notice how we use the Nectar.Repo itself instead of using the FavoriteProducts.Repo, infact beside migration, we won’t be utilizing or starting the FavoriteProducts.Repo, which will help us keep the number of connections open to database limited via only the Nectar.Repo

the view file: index.html.eex

In both of the files we refer to routes via NectarRoutes alias instead of favorite products. To get the assets from Nectar add nectar/web/static to the brunch config’s watched folders:

and in app.js, initialize the nectar code:

Finally for adding the route to Nectar, update nectar_extension.ex with the following code:

And add to install.ex the call:

Now we can see the added routes

let’s update the layout as well:

Starting the server to preview the code

In the previous attempt we were directly running the Nectar server, However since we are essentially working from ground up, let us make another change and add a forward from favorite_products to nectar.

In favorite_products/web/router.ex:

All the usual caveats for forwards apply here. Before doing so please ensure that nectar is added to list of applications in favorite_products.ex.

Note: We have disabled the supervisor for Nectar.Endpoint to specifically allow this and suggest all the extensions do this as well once development is complete. More on this later but suffice to say two endpoints cannot start at and we are now running nectar along-with favorite_products extension.

Now we can run our mix phoenix.server and go about marking our favorites with nectar layout and all.

Layout Present

Testing

We are almost done now. To ensure that we know when things break we should add a few tests of-course, we need to make sure that Nectar migrations are run before running the migrations for favorite products and we need the Nectar repo running as well.

for the former let’s update the test_helper.ex with:

And now to write the tests

The tests for code injection:

Create a new test file tests/model/user_test.ex:

We can test for user_like just like any other ecto model. Let’s skip that for now.

Running Them:

Bonus Guide: Creating your store based on NectarCommerce

We already did this, when we were creating favorite_products extension. A forward to Nectar is all it takes. You can create your Phoenix application and add a forward to Nectar.Router to run your user application. Some extensions might require to be added in the application list for their processes to start, in such cases we need to add a dependency here as well. You might want to do that anyway to support exrm release properly(We will be covering this in our next post).

Suggested Workflow

We can now see developing extensions is not very different from building our store with custom functionality based on NectarCommerce. You start with your store and extract out the functionality into self contained applications and load them back as extensions into Nectar.

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Learning from failures: First Experiment at NectarCommerce Extension Approach

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

Learning from failures: First Experiment at NectarCommerce Extension Approach

Where we left off

In the past few posts we have learned how to write code that extends existing models, routers, added methods to override view rendering and run multiple phoenix application together in the same umbrella project. Let’s Continue to build upon that and write our first extension for NectarCommerce favorite products and ultimately our store based on NectarCommerce.

A layered guide to NectarCommerce extensions

Index

  1. Setup
  2. Model Layer
  3. View Layer
  4. Testing

Setup

Create a new phoenix application to hold the favorite products application, in your shell run inside the umbrella/apps folder:

We could have gone with a regular mix application, but Phoenix/Ecto will be handy in this case, since we want to have views to display stuff and a model to store data.

While we are at it let’s configure our dev.exs and test.exs to use the same db as Nectar, we could write some code and share the db settings between Nectar and our extensions, see: running multiple elixir applications together for more details. But, for simplicity’s sake we are just copying the settings from Nectar to get started.

DB_SETTINGS:

We need to let the extension manager know that this application is an extension for Nectar.
Update the dependencies in extension_manager/mix.exs with the favorite_products dependency.

That should be enough to get us going.

MODEL LAYER

We want a Nectar user to have some products to like and a way to remember them, we can use a join table with relations to user and products to achieve this. let’s generate the model:

Now to point to correct Nectar models. Open up the source and change the associations from favorite products model to Nectar models. In the end we have a schema like:

Of, course this is only the extension view of this relationship, We want the Nectar user to be aware of this relationship and most important of all, we should be able to do something like Nectar.User.liked_products(user) to fetch the products liked by user.

Time to call our handy macros written earlier to perform the compile time code injection. Let’s create the Nectar_extension.ex file in favorite_products/lib/ directory and place this code there:

Don’t forget to update the install file in extensions_manager.

Now we have a user that can like products and product from which we can query which users liked it.

Time to play with what we have built so far, start a shell in Nectar folder with:


Oops!, forgot the migration, remember we shared the db config earlier let’s put that to use and run:

Which will migrate the user_likes table onto the original Nectar database. Back to our shell

Voila! we can now save and retrieve records to a relation we defined outside Nectar from Nectar models without actually modifying Nectar code.

VIEW LAYER

Now that we can save the user likes, we should probably add an interface for the user to like them as well. Which leads us to the first shortcoming, in our current approach, we can replace existing views but right now we don’t have anything for adding to an existing view(Please leave us a note here if you know of a clean & performant method to do this). I also suspect most of us will end up overriding the existing views to something more custom then updating it piecemeal via extensions. For now let’s have a page where we list all the products and user can mark them as liked or unlike the previously liked ones.

controller

Notice how we use the Nectar.Repo itself instead of using the FavoriteProducts.Repo.
In-fact besides migration, we won’t be utilizing or starting the FavoriteProducts.Repo, which will help us keep the number of connections open to database limited only to the Nectar.Repo.

the view file: index.html.eex

In both of the files we refer to routes via NectarRoutes alias instead of favorite products.
To add the route from Nectar, update nectar_extension.ex with the following code:

And add to install.ex the calls:

Now we can see the added routes from Nectar:

So far so good, we have modified and added routes and controller to Nectar’s router. Time to see our handiwork in action, start the server from Nectar application with:

And, visit 127.0.0.1:4000/favorite and click on mark to like a product.

Missing Layout

But things don’t seem right do they? Our Nectar layout has been replaced with the default one used by phoenix. Let’s rectify that.

Update layout_view.ex as:

and recompile and restart the server

On our next visit:

Layout Present

Much better.

Note: When we need to change the extension code while running the server we will have to recompile and reload. We don’t have anything in Nectar right now to monitor all extensions file and do an automatic compilation and code reload.

Testing

We are almost done now. To ensure that we know when things break we should consider adding a few test cases. For that we need to make sure that Nectar migrations are run before running the migrations for favorite products and we need the Nectar repo running as well.

For the former we could update the test_helper.ex with:

But things are not so smooth for accessing the Nectar Repo. Which brings us to what we think is the ultimate downfall of this approach:

An Untestable soution

Ideally, running mix test should work and we should see our test running green, unfortunately this requires Nectar to be compiled before running the tests, which is impossible since Nectar depends upon the extension_manager to be compiled which depends upon all the extensions to be compiled, resulting in a cyclic dependency. Also we used Nectar’s repo for all database work. That works because we were running our server through Nectar and the repo was started in Nectar’s Supervision tree. Which again adds an implicit requirement i.e. Nectar application is available and ready to be started during test time. We could replace Nectar.Repo with FavoriteProducts.Repo using compile time conditionals i.e. when MIX_ENV==test, but it is a can of worms we would rather avoid right now.

This seems like the end of the road for this approach. Where we are failing right now is making Nectar available to extensions as a dependency at compile time and in turn test time. So that they can run independently. Let’s try that in our second approach and reverse the dependency order.

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Extension Approach Explained

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

What will be NectarCommerce

Off-the-shelf Opensource E-commerce application for building an online store.

Provides an Extension Framework to support features not included in core as extensions.

Strives for unobtrusive parallel development of NectarCommerce and Extensions

NectarCommerce is committed to providing a ready-to-use e-commerce solution but the definition of 100% is different under different business domains. It aims to solve common use-cases as part of the project and relying on extension framework to tap the rest.

NectarCommerce Extension Approach

This post aims at documenting how the approach we took to make NectarCommerce extensible is structured, the order of dependencies amongst the various nuts and bolts of NectarCommerce and its extensions as implemented.

NectarCommerce can be divided in two components, both of which reside in the same umbrella application.

  1. Nectar: The phoenix project, which acts as the backbone of your E-Commerce application.
  2. Extension Manager: A plugin manager/DSL provider for Nectar. It also owns the compile time step for Nectar. We intend to make the DSL/Compiler as a separate package and make Extension Manager solely responsible for downloading and installing extensions.

Application Overview

Nectar

Nectar is a run of the mill phoenix application, with models, controllers and some conveniences built into it which makes it amenable to extension. It has no dependencies on other application in the umbrella besides worldly which once done should also be a separate package.

  1. Nectar.Extender, if any module is using this during compile time, Nectar.Extender searches for whether the module which can act as an extension to the using module has been compiled and loads the extension into it. There is a basic naming convention being followed for doing this: if Nectar.Product uses Nectar.Extender then it will search for module ExtensionsManager.ExtendProduct as the module to use for extension. This is a proof of concept implementation, which we can flesh out later to support fully name-spaced extension modules.
    source.

    Note: This currently limits the point of extension per module to a single module. It is by design to keep things in one place.

  2. Nectar.RouteExtender, Similar to Nectar.Extender except it is specialized for extending routes.

So any extension that needs to modify Nectar, needs to utilize the DSL provided by extension manager, we will go through this in detail when we expand on how to build an extension.

Extension Manager

This is a regular mix application that will be used to fetch the extensions and provides a thin DSL(for more details on this see our posts on Model Extension, View Extension and Router Extension) which can be used to express how changes are to made to Nectar. Also any extension that we need to install will be added here as a dependency and properly hooked into the correct modules (Please see the example implementation for one way of doing this). This module provides 3 components for building and compiling Nectar extensions.

  1. DSL : A simple DSL for declaring components that need to be injected into models/router. See our previous posts for how the DSL looks and behaves.

  2. Extension Compiler: It is a basic compile time step that marks files which are using Nectar.Extender(i.e. use Nectar.Extender) for recompilation so that they pick up any changes made to the extensions. It is currently based on how phoenix compiler works.

  3. Install: Extensions can document how they are to be installed by declaring the code using DSL inside method calls and describing how and which modules to call these methods in. These instructions are then followed to compile the injection payload. If this seems cryptic/vague, please refer to the example implementation of favorite products extensions on how the install file is structured.

Dependencies and Code loading

Extensions Manager & Nectar : Extension Manager does not depend upon Nectar directly(it may be a transitive dependency via the extensions) neither does Nectar depend upon Extension Manager. Nectar searches for modules in the umbrella via Code.ensure_loaded to find if extensions exist. While it’s not ideal and as explicit as we want it to be, it is a pragmatic solution for what it allows us to achieve which is basically a form of mutual dependency.

Extensions & Nectar: Extensions should depend upon nectar. Again this may seem counterintuitive since Nectar will be enhanced via extensions, but ultimately we will need the Nectar dependency for running tests, pattern matching on Nectar structs and models and for building exrm releases(more on this later). After we are done we can always recompile nectar to use the extensions.

This concludes our high level description of how the different parts of NectarCommerce interact with each other. Lets continue and see how we can utilize the above infrastructure for building extensions in our next post.

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Running Multiple Elixir Apps in Umbrella Project

The post belongs to NectarCommerce and Extension Framework Awareness Series

  1. NectarCommerce Vision
  2. Extension Framework Game Plan
  3. Introduction to Metaprogramming
  4. Ecto Model Schema Extension
  5. Ecto Model Support Functions Extension
  6. Phoenix Router Extension
  7. Phoenix View Extension
  8. Running Multiple Elixir Apps Together
  9. Extension Approach Explained
  10. Learning from failures: First Experiment at NectarCommerce Extension Approach
  11. Developing NectarCommerce Extensions
  12. Building an exrm release including NectarCommerce

What will be NectarCommerce

Off-the-shelf Opensource E-commerce application for building an online store.

Provides an Extension Framework to support features not included in core as extensions.

Strives for unobtrusive parallel development of NectarCommerce and Extensions

NectarCommerce is committed to providing a ready-to-use e-commerce solution but the definition of 100% is different under different business domains. It aims to solve common use-cases as part of the project and relying on extension framework to tap the rest.

Elixir Umbrella Project

Elixir Umbrella Project is a convenience to help you organize and manage your applications. Applications inside the apps directory are still decoupled from each other. Dependencies between them must be explicitly listed. This allows them to be developed together but compiled, tested and deployed independently if required.

As NectarCommerce extensions are envisioned to be used with Nectar and would be dependent on Nectar directly or indirectly, we chose umbrella project to have Nectar, extensions_manager, user_app, and extensions apps as part of it.

We plan to use phoenix app for extensions where web-component is needed and which would be part of the umbrella project.

The user_app would be a resting ground where Nectar and extensions are evaluated for compile time requirements and prepared to be used together.

As of now, user_app has following responsibilities:

Running Multiple Phoenix Applications together in an umbrella

In the end, a Phoenix application is just another OTP application with a few modifications and configuration tweaks we can expect a couple of them to run together in the same umbrella project. We just need to ensure the following :

  1. They share the same configuration for database. While not mandatory if all the applications are utilizing the same databases, keeping the configuration in one place is a good practice. We can create a shared_config.exs at root of umbrella with some code like this:

    config :shared, :db_config,
       username: "shared_username",
       pool_size: 200
    

    And, then in each individual application at the end of file, use:

    shared_db_config = Mix.Config.read!("../../shared_config.exs")[:shared][:db_config]
     config :app, App.Repo, shared_db_config
    

    Obviously this is use case dependent, some applications may not share a database and only rely on each other for method calls.

  2. Starting the phoenix server. If we look closely at the application file for our phoenix app, we can see a supervisor for endpoint, this starts the phoenix server and binds it to the port.

    We can either start the application on different ports and use an external server like nginx to dispatch on appropriate port based on business requirements.

    We can also use forwards at the end of route files, using the forward "/", App.router. from one of the application and start the server from there.

    If we go with the latter approach we need to make sure that only the endpoint supervisor of current application is in supervision tree, else we will get an address already in use error.

  3. Sharing static assets, we can config brunch to watch the static folder of other applications as well.

  4. Ensure that if using the single server approach, all other applications are listed in the dependency of the main/master application. This will ensure they are compiled beforehand and can be freely used throughout the main application.

Phoenix Framework commands aware of Umbrella project

  • Forwarding Requests to other Phoenix.Router
    • forward "/", Nectar.Router
    • All paths that match the forwarded prefix will be sent to the forwarded plug.This is useful to share router between applications or even break a big router into smaller ones.
  • DB Migration, mention the Ecto Repo
    • mix ecto.migrate -r Nectar.Repo
    • mix ecto.migrate -r ExtensionApp.Repo
  • Listing Routes, mention the Phoenix Router
    • mix phoenix.routes Nectar.Router
    • mix phoenix.routes ExtensionApp.Router

Few Tricks

  • Using View Layout of Nectar from extensions
    • defdelegate render(template, assigns), to: Nectar.LayoutView
  • Using Routes of Nectar in extensions
    • alias Nectar.Router.Helpers, as: NectarRoutes
    • NectarRoutes.favorite_path(conn, :index)
    • alias are resolved at runtime, so routes can be used but compile-time checks can’t be used :(

Our aim with these posts is to start a dialog with the Elixir community on validity and technical soundness of our approach. We would really appreciate your feedback and reviews, and any ideas/suggestions/pull requests for improvements to our current implementation or entirely different and better way to do things to achieve the goals we have set out for NectarCommerce.

Enjoy the Elixir potion !!


Privacy Preference Center