Middleware gets purged after API reload

I’m using the setup from Docker Quickstart and did the following changes to persist my state and load the middleware:

mongo:

    volumes: 
        - "./mongo_data:/data/db"

gateway:

volumes:
    - ./tyk.conf:/opt/tyk-gateway/tyk.conf
    - ./tokenInjection.js:/opt/tyk-gateway/middleware/tokenInjection.js

Now, the first thing that confuses me is that I was able to create an API from the Dashboard although in the documentation it sais that for the CE either the REST API or files must be used. Anyway, it worked so I’m not going to complain :wink:

The next thing that confuses me is that the documentation states that in the CE you have file based api definitions. Since I created the API with the dashboard there are no files (I guess the same holds if you use the REST API to create an API definition). Therefore I went ahead and modified the API definition straight in the Database.

...
"custom_middleware" : {
    "pre" : [ 
        {
            "name" : "tokenInjection",
            "path" : "middleware/tokenInjection.js",
            "require_session" : false
        }
    ],
    "post" : [],
    "response" : []
},
...

However, if I reload the API in the Dashboard my changes get purged and it looks like this again:

...
"custom_middleware" : {
    "pre" : [],
    "post" : [],
    "response" : []
},
...

So, how do I get to enable my middleware component?

Hi Rob,

Are you still using version 2.2 of Tyk? In older versions of Tyk, it wasn’t possible to process an update made to the Middleware from the Dashboard. This would have meant that any changes that you made to the API definition from within the Dashboard would have automatically overwritten any Middleware settings that already exists on that API definition. This should no longer be the case in the latest version of Tyk as it can now process requests made relating to Middleware.

To solve this problem, it might be worth upgrading if you haven’t done so already. Instructions on how to do this can be found at this link.

Kind regards,

Jess @ Tyk

Hi Rob,

Just to add to what I said earlier, there is actually a way that you could make changes to the middleware from within the Dashboard without upgrading to version 2.3. This documentation should explain the process but let me know if you have any questions.

Kind regards,

Jess @ Tyk

Hi Jess,

thank you for your reply. Yes we’re still running version 2.2. Since I am still evaluating I can upgrade to 2.3. For potential productive use that might not be possible that easily anymore.

I tried your solution from the documentation and it does not seem to be loaded. I don’t have Tyk Pro so I guess that guide does not apply to my situation?

folder structure:

root@ee6c5d9ad0a0:/opt/tyk-gateway/middleware/06b81b75f51a4f5852721dd22b3b62fe/pre# ls
tokenInjection.js

and the middleware is just copied from the guide:

var tokenInjection = new TykJS.TykMiddleware.NewMiddleware({});
tokenInjection.NewProcessRequest(function(request, session) {

    log("This middleware does nothing, but will print this to your terminal.")

    // You MUST return both the request and session metadata
    return tokenInjection.ReturnData(request, session.meta_data);
});

edit:
I upgraded to Tyk 2.3. It is still not clear to me how I’m supposed to change the API definitions made from the Dashboard. I edited it straight in the database, is that the way to go? Because even though the definitions I changed in the database do not get removed now, I still can’t see any of the log statements from the middleware.

Hi Rob,

You should be able to add Javascript Middleware using the same methods as you woudl have done in version 2.2. The main difference now is that changes made to any of a specific API’s attributes from within the Dashboard will no longer wipe existing Middleware settings on the API definition itself (which is what happened in previous releases of Tyk). I would advise that you use API requests like the ones you’ve used up until now to add Middleware settings however, the method outlined in the Tyk Pro documentation I linked to last week should also work.

With regards to your missing log statements, have you tried replacing log with console.log in your middleware settings? I suspect that there may be a typo in the docs.

Kind regards,
Jess @ Tyk

Hi Jess,

My question is, what is the recommended method to add middleware to an API in the CE?
In the docs it states that in the CE it needs to be done in the api definition file. However, there is no such file, since the API was created in the Dashboard. So, what do I do?

The link you posted earlier is for the PRO edition, which I do not have. If I try it, nothing happens. I even enabled debugging in the /etc/init.d/tyk-gateway` file to:

program=/opt/tyk-gateway/tyk
args='--conf=/opt/tyk-gateway/tyk.conf --debug'

But I can’t see any output whatsoever.

What I tried is to modify the API definition directly in the database, like so:

"custom_middleware" : {
        "pre" : [ 
            {
                "name" : "sampleMiddleware",
                "path" : "middleware/sampleMiddleware.js",
                "require_session" : false
            }, 
            {
                "name" : "tokenInjection",
                "path" : "middleware/tokenInjection.js",
                "require_session" : false
            }
        ],
        "post" : [],
        "post_key_auth" : [],
        "auth_check" : {
            "name" : "",
            "path" : "",
            "require_session" : false
        },

But is that even possible to do?

Hi Rob,

The recommended way to add your Middleware settings is via a PUT request made to the relevant API definition. In order to do this, you will need to:

  1. Make a GET request to the API definition itself (documentation on how to do this can be found here if you’re unsure)
  2. Copy and paste the response body into the payload of the PUT request that you want to send
  3. Change the custom_middleware object in your current PUT request to contain your Middleware settings
  4. Send the PUT request to the API definition

As for the link to the Tyk Pro documentation, when the file was written, I believe it was assumed that only Pro users would use the Dashboard in this way and so it is actually specific to the Tyk Dashboard. The instructions in that file should still work if you’re using the Dashboard with a single Tyk node. If you’re still unable to use said instructions however, then I would suggest you change the Middleware via an API request as stated earlier.

Kind regards,
Jess @ Tyk

There’s a slight confusion here as to what CE is and what Pro is, I’ll clarify here:

  • CE is just the gateway, configured using files, and no dashboard - you can still get analytics, but you need to use your own BI solution and Tyk Pump to get the data into your DB of choice, CE is a completely open source option, there are no closed-source components
  • Pro has a free tier, where you can have 1 gateway with the dashboard, as soon as you are running a dashboard, you are in the territory of the Pro documentation, anything over 1 gateway gets you into the paid-for Pro territory where you need a license.

So @jess is correct in recommending the folder-based loader as it is by far the fastest and easiest way to debug and run your JS MW.

In v2.2 of the gateway (v1.2 of the dashboard), the middleware component of the API definition was not saved even if it had been written to the DB, and so vanished when you modified it via the API/DB and then edited via the dashboard.

In v2.3 this restriction has been removed, and the middleware block can be added to the API definition with the REST API, but there is still no UI for it.

In v2.3 there is also the option of using an asset server, this server can then host “bundles” that your gateway can fetch, verify and unpack into the coprocessor, there’s a CLI tool too, to help put them together and the docs are in all the tutorials.