During the past weeks I have spent about 2 hours per week on mentorship sessions with various developers from the Devz Community. I am really thankful for everyone that chose me among all the mentors, and I took notes on most of the questions.

I immediately found frequently asked questions for all the topics that I listed, so I decided to write a blog post with them. In the future I plan to update this post with more questions as they come.

Technical Leadership

I Feel I Am Programming Less And Less [As A Tech Lead], Is This Normal?

Yes. Technical leaders are moving to a path where you are no longer just producing software i.e. an individual contributor. This is probably the most asked question regarding transition from senior developers to tech leads.

What’s important here is a bit of time management skills. I suggest having a balanced schedule allowing new tech leads to still program while considering the other skills required. Here is an ideal example:

• 30% of your time could be spent on the mentorship of your team members,
• 30% could be spent on meetings/product development,
• 30% could be spent on coding,
• 10% can be spent on growing your own skills.

A good reference for this kind of transition for me has been the work of Patrick Kua on the topic. Here is the most influential talk I could find for this.

What Skills Do I Need to Be A Tech Lead?

I think there are fundamentally 4 skills that you need to develop:

1. Leadership skills, being technical or non-technical,
2. System Design and Architecture skills and,
3. Developer skills, which you may already have.
4. Time management skills to balance your time spent on these.

I Want To Keep Coding But Grow My Career. Is There A Way To Do That?

I think this entirely depends on your organization. If the only path for growth to happen is go into management, probably it will conflict with this particular interest of yours.

I have experienced this myself and usually Tech Leader positions conflict with time spent programming. If your particular organization has technical paths (such as Principal Engineer) you can try to move into that direction, but most companies do not have such paths.

You can always trailblaze the path by communicating clearly with your managers the pursuit of such goal. Some companies can accommodate you but others have different priorities thus they will conflict with yours.

Architecture

As An Architect, Should I Still Develop Software?

Yes. Undoubtedly you don’t want to become the proverbial Ivory Tower Architect. If you detach from programming completely, most probably your solutions will also be detached from the code teams produce and thus it becomes counterproductive because developers will be forced to retrofit your solution instead of fit it more organically.

What Coding Tasks An Architect Should Focus On?

In my experience, the best architects I have worked with usually spend time on proofs of concepts and foundational technology. I have worked with architects that usually pair program with developers to unravel solutions that can fit the actual working code instead of coming up with solutions that may or may not fit the current architecture.

I think this approach has the best results overall because it proofs that your architecture guidelines and implementations fit into the software other teams are developing.

How Can I Know If My Architecture Is Successful?

Fitness Functions can be used to measure architecture goals. They also can be used to drive evolution to certain parts of the overall system once you know what to aim to.

An example of this is a fitness function that returns either the time, or the number of commits between your production code and the latest commit in your development environments. This function will let you know if you are properly applying Continuous Delivery.

Another example is Dependency drift fitness function. It allows you to understand the Drift between microservices regarding libraries and dependencies which can be interpreted as an indicator of stagnation/ossification of services.

Microservices

What’s The Point Of Microservices?

This is a tough question, but I usually focus on these points that work well with the evolution analogy:

1. Microservices are about change independence. As an example of a evolutionary architecture, microservices allow evolution on different parts of your system quickly. Some services will be very stable once they reach a certain point but others need to often change to accommodate business needs. Again, some parts of the organism evolve while others remain stable.
2. Faster time to market. Following from the previous point, once you know that the microservice architecture style allows for faster change and independence you are able to quickly prototype new ideas because you can start new business lines from scratch quickly, with the right tools (spanish).
3. Making innovation cheaper. As a side effect of the previous point, microservices enable to quickly prototype new ideas because you can start new business lines from scratch quickly with the right tools.

Do You Think We Are Ready To Adopt Microservices?

Microservices are not a free meal. Conway’s law is an important concern if you plan to adopt this architecture style. This is because there is a social/organizational component to every software architecture style and having organizations simply jump into microservices without the proper mindset can be problematic.

If you use Conway’s law to your favor, service-based architectures can help move faster, but without organizational perspective it usually works against the adoption.

DevOps

What Are The Most Common Questions You Are Asked During Interviews?

After quite a bit of interviews I have seen common questions and follow-ups:

1. What is DevOps? The interviewer wants to know if you “get DevOps”. Focus on why DevOps is more of a practice and cultural mindset instead of titles.

What is SRE? Software Reliability Engineering is kind of defined as:

One could view DevOps as a generalization of several core SRE principles to a wider range of organizations, management structures, and personnel. One could equivalently view SRE as a specific implementation of DevOps with some idiosyncratic extensions.

Per Google’s own book on the matter.

1. What’s the difference between DevOps and SREs? Per the previous question, but paraphrasing here, I think of SRE as an implementation of DevOps practices done by Google.
2. Can you describe/document a pipeline to deliver code from scratch? This is a lengthy question that may deserve its own post! it gets asked often.
3. What are SLAs and SLOs?

I Want To Transition To A “DevOps”/SRE Position. What Should I Study?

Most “DevOps” positions are really infrastructure positions. Most SRE positions are SysAdmin positions. Having said this, if you target infrastructure positions you should focus on:

• Learn how to work with a specific Cloud Provider. This implies learning how to manage and architect infrastructure on a specific provider. AWS has around 30% of the market share as of the time of this post, so it maximizes your opportunity to land a job. This implies Infrastructure as Code practices and tools to help you provision and manage such infrastructure.
• Learn tools and practices regarding Continuous Integration and Continuous Delivery. Most interviews in the infrastructure space ask you to design a pipeline to deliver services. The goal would be to be able to design pipelines using Jenkins pipelines, Github actions, Gitlab pipelines. I recommend watching Ken Mugrage talk about modern CI/CD practices, then learn a specific tool.
• Learn about Observability tools and practices. A good rule of thumb is to use tools from the Cloud Native Computing Foundation Observability And Analysis landscape. Most interviews, in my experience, will ask you to configure not only instrumentation but also alerts and dashboards for services and infrastructure.

If you are targeting sysadmin positions, they probably will overlap with the previous points but also require the following:

• Networking fundamentals and Software Defined Networking.
• Operating System maintenance and provisioning (package management, configuration).
• Container/VM Orchestration Tools (Kubernetes is the most popular nowadays).

I recently wanted to introduce my daughters to programming, so I decided to use some kind of sensor to prototype a small application and teach them how to make hardware and software work in tandem as I believe having something physical would be more interesting than me typing on a REPL.

Now, I knew a raspberry pi had a way to connect sensors using its General-purpose input/output pins, so I decided to build a non-trivial application to get used to the programming model and at the same time have experience to build something more complex later.

Hardware Required

To start the project I did some research for sensors and while there were quite a bit of choices, I got interested in the KeyEstudio CCS811 Carbon Dioxide/ Air Quality Sensor for Arduino which is compatible with Raspberry Pi 5V pins. It also works using the I²C communication bus which is also supported.

Now, in order to prototype faster, I got a T-type breakout, a solderless board, and rainbow cable, plus jump wires. With this at hand, I was able to start interfacing the GPIO with the sensor.

Wiring It All Together

Now, it was a while (since my college days) that I had work on a solderless board but fortunately there is a diagram to connect the sensor to an arduino board thus you can deduce how to wire it to the equivalent pins on a Raspberry Pi.

With this diagram, and the official Raspberry Pi documentation, I was able to find the correct pins without much issue. For reference, here is the Raspberry Pi GPIO pin diagram (this is for Raspberry Pi 4 Model B).

Finally, with the help of the T-type breakout, and the solderless board, here is a picture of the connections. Note that the breakout already labels the pins so is not hard to match them.

Validate Connections To Raspberry Pi with i2cdetect

Now, with everything wired, the fastest way to detect if the sensor is properly interfacing to the I²C bus is to run a utility which can be installed on the Raspberry Pi Os named i2cdetect. Here is the command:

pi@elderserver:~ $ i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- 5a -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

The way to interpret the output of the command, and the parameters is the following:

• i2cdetect takes a positional argument named I²C bus, in this case it maps to a linux device by number depending on your board. On the Raspberry Pi 4, it is the device number 1 located at /dev/i2c-1. The -y option is to disable the interactive mode.
• The output of command represents the bus addresses in which the detected components can be found. In this case, the only address responding is 5a, which matches the vendor specified address for the CO2 sensor. You can find more details in the datasheet of the CCS811 sensor.

All I²C transactions must use the (7 bits) slave address 0x5A or 0x5B depending on status of ADDR pin when writing to and reading from the CCS811.

Read Data From The Sensor

This was the tricky part of the project as the documentation in the wiki of the seller had mostly C code which somehow was factually incorrect once I compared it to the official programming and interface guide. I struggled about a day using the wiki, so I finally started to search for the sensor documentation instead of Arduino IDE examples.

TypeScript, I²C And Prometheus

The full source code can be found in this Github repository.

In order to work with I²C sensor I decided to use the help of the Node ecosystem using a small library called raspi-i2c which allows sending and reading from devices attached to the bus in a very procedural way, with sync and async functions.

To build the web service itself I used Express to handle the HTTP requests coming from Prometheus and a small library called node-exporter-prometheus to generate the Gauge metric type to expose the sensor data.

Programming The Web Service

The Express Server

An Express web service is not really that difficult, the relevant part is here:

const app = express();

app.use(promExporter.middleware);
app.use(readSensorMiddleware(i2c));
app.get('/metrics', promExporter.metrics);

app.listen(serverPort, () => {
    initSensor(i2c);
    console.log(`server started at http://localhost:${serverPort}`);
});

We basically create an Express application, then register a middleware to gather the common metrics using node-exporter-prometheus and register our own middleware to read the sensor before finally exposing an endpoint under /metrics.

Initializing The Sensor, A Quick Premier Of I²C

The initSensor function will apply the initialization logic required for the sensor to start reading the environmental data. It follows the following diagram:

For this, the raspi-i2c library provides two basic methods to interact with the I²C sensor:

1. writeSync(address: int, register: int, buffer: Buffer): void this method allows writing to I²C devices by their addressed, to a specific register.
2. readSync(address: int, register: int, length: int): Buffer this method allows reading a specific register of I²C devices by their address.

These are the two methods required to initialize the sensor. As it can be inferred, the I²C sensor is very simple to interact with. As long as you have the sensor address (in our case 5a), and a register to read/write from, you will be able to initialize and read the required data.

This small snippet allows to read the hardware id from the sensor and check is the correct one:

// SENSOR_ADDRESS is 0x5a, HARDWARE_ID_REGISTER is 0x20
const hardwareIdBuffer = i2c.readSync(SENSOR_ADDRESS, HARDWARE_ID_REGISTER, 1);
const hardwareId = hardwareIdBuffer[0];

if (hardwareId !== SENSOR_HARDWARE_ID_MAGIC_NUMBER) { // This is 0x81
   console.log("Hardware ID did not match: ", hardwareId);
   // ... call error handler
}

As you can see, this is very straightforward. You choose either if you want to read or write from a sensor and how many bytes. You pass a Buffer or get one depending on the operation. After that is just a matter of inspecting the bytes returned according to the datasheet.

Read Data From The Sensor

Finally, the readSensorMiddleware will read both the CO2 and TVOC output from the sensor data address and set the Prometheus Gauges along any errors found. Here is the relevant snippet of code:

// SENSOR_ADDRESS is 0x5a, STATUS_REGISTER is 0x00
const statusRegisterReading = i2c.readSync(SENSOR_ADDRESS, STATUS_REGISTER, 1);
// bit at position 4 signals new data is ready
const isDataReady = bitwise.integer.getBit(statusRegisterReading[0], 4);

// If data is not ready check errors
if (isDataReady === 0) {
    checkErrorRegister(i2c);
    return;
}

// Read Data, 8 bytes, the first two bytes have the co2 reading
const buffer = i2c.readSync(SENSOR_ADDRESS, RESULT_DATA_REGISTER, 8);

// Convert the first two bytes to a 16 bit integer with big endianess
const co2Reading = buffer.readUInt16BE();

// Set the Prometheus Gauge to the result
co2Gauge.set(co2Reading);

Endpoint Result And Grafana

Now, with everything wired together, here is the resulting payload when sending an HTTP request (omitting comments and other metrics).

$ http --body GET http://192.168.1.20:8080/metrics
co2_ppm{appName="co2-sensor-pi"} 405
tvoc_ppb{appName="co2-sensor-pi"} 0
message_invalid_errors{appName="co2-sensor-pi"} 6
read_reg_invalid_errors{appName="co2-sensor-pi"} 0
meas_mode_invalid_errors{appName="co2-sensor-pi"} 0
max_resistance_errors{appName="co2-sensor-pi"} 1
heater_fault_errors{appName="co2-sensor-pi"} 1
heater_supply_errors{appName="co2-sensor-pi"} 2
unknown_errors{appName="co2-sensor-pi"} 2

Once we have Prometheus scrapping the metrics, here is the resulting Grafana dashboard:

Previously in my post about Digital Platform Accelerators, I wrote about Project Kickstarters. In this post, I will try to get deep into the patterns I have seen and implemented.

In many companies I have worked we usually implement authentication and authorization, logging, telemetry, etc. I have implemented these in many project templates as a means of showing the capabilities of the platform on which the software runs and the fundamental structures that it provides to layout cross-cutting concerns. The idea behind project kickstarters is that they already give you an implementation of these concerns that is ready to be used and allows developers focusing into implementing business logic.

Reasoning: Reference Architecture

Many of the problems I have had with Software Architects is that whenever I have worked in a big company with many teams, architecture is usually retrofitted not implemented. The proverbial Ivory Tower Architect is one that gets a cross-cutting concern and then thinks of the solution without taking a look at the code that is already implemented. Thus their solutions mechanically clash with the existing interfaces.

Good reference architecture, on the other hand, starts with solutions and research. Proofs of concept are usually a good starting point to generate templates because they are necessarily implementing something that we want to reach. If by some reason we cannot implement a solution to a problem, it means that our architecture is not fitted for our context.

I have also met really good Software Architects and a common pattern I have seen is that they always say “let us prove that this is possible, that we can implement it” on one way or another. Good project templates serve as reference architecture because you can see them deployed and working on a Software Platform, visit their repositories and understand how they integrate to the platform.

Project Templates

Project templates are not a new concept but most probably they became more necessary with the advent of the microservices architecture. If you have a Digital Platform, good architecture mandates a series of cross-cutting concerns that conceptually can be grounded into templates that should require to do minimal customization to start the development of a new service.

I have seen many patterns for these but the most common is a single git repository in which architects and devs alike pour all the best practices they have learned for development along with sample integrations with technologies within the Digital Platform such as authentication or telemetry. These repositories usually are the product of spikes to know if certain technology can be a good fit for the projects or work that has been done previously.

Nonetheless, project templates are the lowest common denominator to use as Platform Accelerators because they require a lot of customizations to actually become an independent microservice. Cloning a project and do a lot of renaming is probably an error-prone task as the smart reader will notice.

Project Generators

Once you have achieved enough maturity regarding project templates, the next logical step is to create tooling that uses these templates and produces ready-to-use projects.

The focus of this post is to talk about them in the context of Digital Platforms so tools like Micronaut or Gradle initializers are not going to be described because, as far as I know, they do not allow you to customize the projects they already produce with features specific to your context.

Tooling available to create project generators is varied. I have used mostly Yeoman and Atomist. But there is also Maven Archetypes and Micronaut mn utility, Gradle init subcommand, etc. Because the focus of this article goes along the lines of Digital Platforms, tools that do not provide a way to integrate platform-specific features are not discussed e.g. Micronaut mn.

Yeoman

Yeoman is a project scaffolding tool similar to Maven archetypes or Rust Cargo. In principle Yeoman is really easy to use as it works with templates and an “in memory” file system that allows you to stream files from a predefined location and apply transformations to them.

Whatever you can do with memory-fs editors you can do with all the files using Yeoman. I have also used sub-generators as feature toggles to allow generation of code with specific features on them.

The drawback of using templates is that you will not have a good feedback cycle unless you generate a project from scratch and then run the generated project and hopefully run a good test suite contained within the project. For example, if Yeoman generates a front end application you will want yo run npm run buildor if you are generating a Gradle project then running ./gradlew build. These will allow you to know if your code compiles and their tests are running fine with the set of features you want. All of this takes time so I would not recommend templating files at all.

What has worked best for me is to download a repository into the directory where Yeoman is expecting the template files to be and then do the replacing and renaming. This ensures that at least the starting point of your code generation is a project that hopefully is already in a good state (by having this project have a pipeline and a battery of tests that actually work).

The following problem here is making sure that the generated project is not in a bad state and soon enough you will have to have a pipeline that generates and discards projects after running tests on them.

Finally, the tools Yeoman gives you are very basic. String replacing, file renaming and so forth are the bare minimum for project generators and soon enough I had found a lot of complexity while trying to make more sophisticated things such as properly remove/add features to a project depending on the user needs. You could theoretically create projects that have only the features some devs will need but this becomes a complex problem due to the number of permutations that happens with these projects.

Atomist

Atomist approach to code generation takes the idea of independent seed projects that you take and apply code transformations to them. This is no templating mechanism so Atomist proposes using Microgrammars and AST transformations to automate the changes to the project.

While I agree that you could use other AST transformations using Yeoman directly using Vynil streams ASTs, I do think that the API Atomist provides, along Microgrammars are more simple and powerful to use as they provide really good abstractions over code projects and code transformations.

Using a Software Delivery Machine, you can use Git repositories as the starting point for a new project. By applying code transformations to the source code, you will produce a fresh project. The SDM will orchestrate the transformations and produce a new Git repository and also can push it to a service such as Github or Bitbucket ready to be used.

In a later post, I will discuss approaches to code project generators and probably code examples to create feature toggles using them.

Previously in my post about Digital Platform Accelerators, I wrote about Distributed Refactoring Tools. In this post, I will try to describe the different tools I have used and the circumstances that lead me to use them. Also, why they are necessary.

I have worked on four microservice projects on different roles. From front-end and backend developer, tester, infrastructure developer, and a mix of all of them. While I worked on three different companies during those projects, I almost always found the same problems appearing.

The need to share code between services using platform libraries and the consequent maintenance that comes with them. The need to create policy through pipelines that evolve constantly and end up breaking project builds due to backward-incompatible changes. And finally, the need to update configurations for both pipelines and microlibs on several projects at a time.

In hindsight, all these problems mostly come from the same source: drift. Be it configuration drift due to a myriad of properties files or configuration sources; dependency hell due to library version drift; or delivery drift caused by feature parity between projects and the pipelines required to deliver them.

Why Does This Kind Of Drift Happen?

It’s because you share code and settings. Configuration drift happens when you have different settings or initialization data for your services or infrastructure. Dependency drift evolves to dependency hell when you have different versions of the same library and you share those versions on other software projects. Finally, delivery drift happens when your services cannot be delivered with the same pipeline definition over time.

The organic nature of software is what is causing drift. At some point, we have created strategies for conveying meaning on change e.g. versioning and more specifically semantic versioning. With these versions, the goal is to keep up to date with the latest release because it should be the most stable or has the latest features that help us evolve our platform.

The problem with semantic versioning is very misleading regarding two things: it’s not always the most stable and in most programming languages there is no way to enforce it. Even Joe Armstrong, of Erlang fame, tried to find ways to enforce that even a function was exactly the same as other function to enforce compatibility.

The latest release of a library is not always the most stable due to the nature of modifying software itself. Regressions happen, not breaking compatibility with previous releases is a goal but there are no guarantees.

What I think is that semantic version, even if well-intentioned, is not sufficient if you want to avoid drift on your dependencies or configuration. The smart reader has probably also thought about dependency locking. Some tools have already integrated this feature so that every time you build a software project it will always resolve the same dependencies. The problem with version locking is that it poses the problem of maintenance.

Version locking and the organic nature of software conflict with each other. If you lock dependencies, sooner or later you need to upgrade them or they will become obsolete. It’s on your best interest to not be out of date with security patches and bug fixes.

Distributed Refactoring As Means To Ease Drift

Unless you are on a monorepo (which has different challenges, out of the scope of this post), there are many constraints to evolve your software and keep up to date in a Microservices architecture.

The first problem with Microservices is that, unlike the monolith, you need to have platform libraries. The share-nothing approach could be useful but leads to a lot of duplicated efforts and even more drift because everyone will be tempted to solve the same problems with different strategies.

Platform libraries are implementations of cross-cutting concerns at a platform level. With a monolith, AOP used to be the way to achieve this, but you cannot intercept microservice calls without adding more overhead between services. Examples of this are authentication implementations, logging, monitoring, telemetry, etc. Even the pipeline definitions of build servers such as Jenkins are a way of policy enforcing and they require updates.

Now that you have implemented cross-cutting concerns using those libraries, keeping them up to date will be the next challenge. This is where distributed refactoring comes into play.

If you are into a monorepo, refactoring can be a single commit and it’s atomic. Updating libraries in different repositories will not be atomic and it will be difficult to do by hand if you have more than a few microservices to update. Even if updating different repositories is not atomic, automating these updates as much as possible will save a lot of developer hours.

I remember working on a single pipeline to deploy only a certain type of microservices. Every time I had to add a feature, it required modifying at least a dozen of services of that type. That could take anything from a couple of hours to an entire day depending on the definition and the size of the change. Even with shared pipeline libraries, updating the versions used is still a task that requires effort per microservice.

The Tools I Have Used To Execute Distributed Refactoring

Atomist

I have used Atomist at very early stages in my second microservices project. At that time I worked as a tester and we needed to use clients for a huge amount of SOAP services that provided auto-generated code.

Unfortunately, while we could generate code from those services because I was working as a tester, I knew that the service contract was going to change eventually and I did not want to create builders by hand for the myriad of objects that represented the response and request for those services.

Then I found Atomist and I found that I could create software that transforms software, I created a small program that added the required annotations to the generated code.

While this approach helped me a lot, on the way I discovered many other features that Atomist has that have helped me resolve all the problems that I have faced always during the development of microservices.

Atomist fingerprints allow you to keep track of the different versions of the platform libraries being used and with this drive updates. Configuration drift can also be tracked with Atomist. But Atomist goes beyond just tracking, you can create Code Transformations to keep projects from drifting if required with pull requests that developers can accept when ready (or just apply them if you feel empowered to do the change yourself).

Generating pull requests to update libraries using microgrammars to look for specific parts of Gradle build files or NPM package.json files is possible and is very simple to do.

Delivery drift can also be tackled by having Atomist fix/add step definitions and Jenkins pipeline versions to keep projects up to date with the latest policy in your platform.

Finally, I have been experimenting on creating backward compatibility with project kickstarters by annotating new platform features and make Atomist copy such features from seed/reference architecture projects.

This is important because project kickstarters and reference architecture start to drift the moment that you used them to start a new microservice. If users want to use new platform features, in most of the projects we have to do this manually with migration guides and manual steps. If you can automate adding new features, for example, the use of a secret registry such as Hashicorp Vault, you could save a lot of developer hours invested into integrating that feature from reference architecture services.

I have been in only one project where they used Snyk and it solves the problem of dependency version drift but with an approach focused on security. I really do not have much experience using the tool but I can vouch that it helped many other developers to keep dependencies up to date when critical security issues arrive.

Just recently we have had a huge amount of pipelines failing due to security scans made by OWASP and keeping the relevant libraries can consume a lot of time. While most of the time we need to ignore the problem because even the newest version does not resolve it, Snyk is capable of fixing them automatically with patches later on.

I became aware of rewrite only a few days ago and it looks very powerful. But it only works on Java projects. Its API looks fantastic for distributed refactoring and I have seen presentations were it could refactor all the Guava references of a method on GitHub!

I am researching this project for future use as one of my current project goals is to provide compatibility with other projects thus I can see the potential of rewrite to refactor calls to microlibraries we use to provide platform features.

Conclusion

Fighting microservices drift is a daunting task and while I have seen and implemented various approaches to this and it is still very difficult. I believe that keeping team independence is great, automated refactoring possibly could be better.

I remember in the first project I had to go and review all the front-end code of various codebases and create comments for developers to update their dependencies to introduce some features dictated by the platform. My surprise is that while some teams fixed the issue right away, others did not. In this case, I lacked the authority to enforce or apply the changes myself so that went on for a while until things started to fail.

In further projects, I had more power to do this but I still find that going to each repository and apply the fixes by hand is very error-prone and this is not the ideal scenario because drift happens in many places at a time and all of a sudden. That moment was when I started to look for tools that allowed me to refactor code automatically or allow me to speed the process.

My point is that even if you could keep things up to date for a while yourself, the scale of a microservice architecture will reach such a mass that doing things automatically is a better alternative than putting the effort to do it and feel the pain.

Interesting interview. But often a "CI/CD pipeline" actually means 100 or 1000 separate pipelines. So the oil needs to be applied in many places. Hence we need to be able to express team-wide policy https://t.co/PlDFIVmmIj

— Rod Johnson (@springrod) April 9, 2019

Previously in my post about Digital Platform Accelerators, I wrote about Delivery Workflow Tools. A pattern within these tools was Shared Pipeline Libraries.

So far I have implemented pipelines on three different microservice platforms for three different companies and I intend to explain my experience in this post.

Reasoning: Do Not Repeat Yourself And Avoid A Death By A Thousand Cuts

About three years ago I was working on a project that had started with relatively few services, only five of them and three front end applications. By instructions of the CTO at the time, I was assigned to the testing team as a quality engineer in charge of writing Cucumber tests for a series of Web Services made using JAX-WS that were going to be “refactored” into modern technologies.

I am glad to say that I proved my CTO wrong because I wanted to go to the team in charge of building the tooling that these new services were going to use. This team was branded the DevOps team and they were going to be in charge of provisioning the infrastructure and maintain a single Jenkins server that will be used for all of this. By implementing the pipeline for running my own tests against the web services I was able to ask for a chance to work on that team because they were understaffed at the moment.

I was very new to Jenkins so the first thing that we tried to do was to program the Jenkinsfiles for each project ourselves in the eight repositories that were available.

Soon I found that I had a lot of repetition on these files. I mostly had copy-pasted the code there and changed a few things such as project names and other parameters.

As you can see this was not good but at the time the idea was that the teams in charge of those projects were going to write themselves those files once we had set up the patterns to deploy to the infrastructure we provided. Of course, software is organic and more requirements come and pipelines need to evolve thus we had to oil those Jenkinsfiles many times to add new features such as security scans, SonarQube quality gates, etc.

Our developers were busy delivering value features so the work of our team was to introduce patterns for these features and hopefully they will implement them in their Jenkinsfiles. Alright? For the clever reader, yes, we had to also write these in each repository Jenkinsfile.

Jenkins Shared Libraries

Updating every repository with these new features implied that we first worked on one repository at a time by modifying the Jenkinsfile by hand and then run the pipeline to see if it worked. Remember, I was very new to Jenkins and the rest of my team was very much understaffed so they were busy gathering requirements and trying to get order out of the chaos.

I started to research Jenkins shared libraries that will allow me to define the common steps and encapsulate them into a single parameterizable abstraction.

We started to replace many stages in our Jenkinsfiles with a single line of code with the new custom steps. This development outcome was that we finally started to remove duplication from our Jenkinsfiles but our pipelines still look very much alike.

During my last two weeks at that company, I started to implement the whole pipeline as a single step. This required to standardize the projects running such pipeline definitions to use the same tooling and define the same build system tasks.

Testing Steps And Definitions

My next project had Jenkins up and running already and it had a really nice shared library but in hindsight, testing by triggering the pipeline time and time again is not very efficient because you will be waiting a lot of time for the pipeline to reach the steps that you are interested.

What we found is that testing Jenkins shared libraries is a very complex matter because from the beginning Jenkins introduced this feature without any mechanism to test them in isolation. Having to use Groovy is also a challenge as the dynamic nature of the language makes you test double on finger errors such as the number of parameters and their types.

Fortunately, this is not a new problem thus there are projects that ease these steps.

jenkinsci/JenkinsPipelineUnit

Framework for unit testing Jenkins pipelines . Contribute to jenkinsci/JenkinsPipelineUnit development by creating an…

github.com

We extensively used jenkinsPipelineUnit to test our workflows and it helped us mainly to avoid regression bugs but testing the integration with external systems (such as SonarQube, Fortify, Terraform, AWS) still was very time-consuming.

Convention Over Configuration Plus Flexible DSLs

Once I experienced a somewhat mature shared library, I started a new project which required to scale things to a new level as the number of microservices was many times more than I had seen so far. When this situation happens there are always going to be edge cases.

These can be in the form of projects that require to skip steps, projects that require a slightly different task order and various other changes. When you start to see this problem emerging a useful pattern is creating your DSL for pipelines within the Digital Platform of your company.

This means that instead of defining steps with pipeline definitions, you will define a DSL that will be focused on the Platform and following Convention over Configuration to detect the needs of a project.

openshiftPipeline.groovy – Medium

GitHub Gist: instantly share code, notes, and snippets.

This is also a highly conceptual task but I can enumerate a few things that we have had implemented:

• Detect project types and configure the steps that run depending on the type
• Detect requirements of the project from its structure e.g. if you have a file with a database definition, apply automatically a step to provision it within your orchestrator
• Inspect for tasks and skip if they are not found, for example, check if there is a Gradle task for checking style and execute it if needed
• Apply fixes to common mistakes within projects and alert developers early of this problem through notifications. For example, fail/touch files required for the pipeline to run properly like required manifests

All these integrations are very context-dependent thus they will change with your platform. It is important to keep backward compatibility through the support of older models as you evolve the pipeline DSL.

Again, the important part here is that the DSL reflects the standardized way you want application pipelines to behave to support your organization workflow i.e. they become Delivery Workflow Tools.

A Digital Platform is described as the set technologies that function as the context in which the company applications run. A combination of middleware that serves as the foundation for the different products and APIs that are part of an organization.

We have had Platforms as a Service (PaaS) for a while and I have worked on three digital transformations whose objective is to deliver applications and the platform to sustain them.

During those transformations, I have observed Digital Platform Accelerators emerging as patterns and tools that help you to reduce the friction created by deploying and managing many new services into a Digital Platform. The objective is clear: they provide faster access to use the platform itself.

Here are some types of accelerators I have implemented and used many times during these projects.

Project Kickstarters

In order to create services fast, enterprises have created mechanisms to build software applications using conceptual templates that can be deployed to a platform. Such templates are the aggregate of the technologies, best practices, and integrations required to run inside the Digital Platform.

I have implemented a few Project Kickstarters using different mechanisms and patterns and here are some of those:

Project Templates

The idea is to have a project that serves as a blueprint of the typical application deployed in your platform. This project is cloned and after some modifications by hand, you get a repository that can be used to create new services in your platform.

Many times these templates are divided by technology stack. Maven archetypes are a good example of this pattern but a significant difference is that the project templates contain integrations that are required for the application to run inside the platform.

These integrations differ from company to company but the usual cases are authentication, authorization, database integrations, caching. As you can deduce from these, we are talking about cross-cutting concerns that a Digital Platform should provide.

Project Generators

Eventually, you will have as many templates as technology stacks. And these templates will evolve as the platform starts to change to integrate more cross-cutting concerns. The logical next step is to provide platform users with a way to generate applications without the error-prone task of modifying a Project Template by hand.

Enter Project Generators. These applications will take a set of parameters that change between applications in your Platform and produce a repository ready for deployment.

Another benefit is that a generator is software too so it can be used to remove or add features from the generated projects. For example, if the new application does not require a database, the generator can remove the code that provides database access from the codebase.

Feature generation is a complex topic and it’s very hard to implement. Generators tend to grow complex if many features are provided thus is always better to keep them simple or have template projects to pull code from and transform them.

My dislike for tools such as Yeoman is that they work on a very low level by mostly manipulating text files as templates or low-level string replacement. This had lead to many bugs in which replacing a string in many files and over many layers can lead to very nasty bugs that are hard to resolve.

Atomist provides the next step of this kind of tooling in my opinion as it allows you to use AST transformations for code and also string replacements if your needs are not that sophisticated.

Examples of this are Yeoman, Atomist, JHipster, https://start.spring.io/.

Delivery Workflow Tools

In order to be deployed, generators need also a standardized pipeline from your lower environments to production. A typical problem in Digital Platforms is trying to establish this pipeline as your projects may differ in very little details that need to be captured by your delivery pipeline. Here are some useful patterns for achieving this:

Custom Build System Integrations And Tasks

You may have tasks small enough that can be integrated into your build automation tools such as Gradle/NPM. You encapsulate your steps in the tool instead of the pipeline and it gets executed by calling a task defined there.

These tasks usually conform to mandated designs in order to progress the applications through the platform. Examples of these tasks are running isolation and API tests, OWASP dependency checks, publishing to different platform environments, packaging the application in the preferred format for the servers/containers, etc.

A possible example of this tooling is implemented in Netflix Nebula. I also have created custom Gradle plugins that encapsulate many steps required to build applications and deploy them to a platform such as Kubernetes/Openshift.

Shared Pipeline Libraries

CI/CD tools such as Jenkins provide you with mechanisms to create shared libraries that can store common steps required to deploy applications. These libraries are pulled from your CI/CD server such as Jenkins and have definitions of shared steps that are used across your applications. They also can provide templates of fully-fledged pipelines that can consume your source code repositories and deliver artifacts up to production.

Unfortunately, shared pipeline libraries can sometimes be very complex and many developers I know think these are against the DevOps way of thinking as the platform is somehow mandating the structure of your pipeline.

On the bright side, Shared Pipeline Libraries avoid the problem of distributed refactoring inside applications within a Digital Platform. Imagine having to modify the pipeline definitions of every application if you introduce a new stage required for all of them. An example of this is security scanners mandated by your security engineers.

Unfortunately, creating these shared steps and pipelines requires a lot of effort because they need to be as general as possible but eventually you will have small edge cases. Maybe you need a different database in one project. Maybe you need to skip certain stages in another project.

Most sophisticated Shared Pipeline Libraries I have coded use Convention Over Configuration and small DSLs to build standardized projects. This removes the friction from deploying as new teams can have a newly generated project delivered into production in minutes.

Delivery Workflow CLIs

Over time, delivery steps in a pipeline start to get more complex as bash commands/scripts are being abused. In order to use a fully fledged programming language, a CLI tool is created that is called from the pipeline instead to execute more complicated stages.

The workflow of the application delivery and integration is captured in these CLIs and they sometimes mirror the stages of the pipeline. The benefit of this is that you are decoupling your delivery process from the CI/CD thus you could potentially run the CLI locally and achieve the same result, helping you with debugging and get faster feedback than waiting for the pipeline to run.

Another benefit is that the CLI can accommodate different technology stacks while keeping the complexity low because you are not tied to shell scripts.

NEWT (Netflix Workflow Toolkit) is a possible example of this pattern.

Shared Build Images

If you have created a standardized pipeline or workflow CLI chances are that all the required tooling is scattered in different places. For example, if you are using Jenkins you will have Jenkins agents for each technology stack or even for each tool.

This becomes a maintenance burden sooner or later for your DevOps/Infrastructure team thus a pattern I have seen emerging is to accommodate the bulk of your tools into a single Docker image/VM Template. If you only have a single place to add tooling and a single image to use in your CI/CD server you only will have a place to modify to add tooling required for new platform features.

This is important as many tools such as security scanners and tools to deploy applications such as kubectl are required regardless of the technology stack. You only pay the price of building this tool once (even if it is a lot of time) but you can easily update agent definitions and orchestrate pipeline changes that use the required tooling.

Distributed Refactoring Tools

Unless you are working with a monorepo, if you introduce a feature to your Digital Platform you will have to look for their dependents and update them accordingly to use your new features.

An example of this is typically when there is a need to introduce new stages in your delivery pipelines such as security scans. If you have a shared pipeline library or a CLI you might only need to update their versions but this has to be done across all the projects that use your pipeline and/or CLI.

Project Generators and Templates also only provide the first steps for your project. As they are updated with new features, projects that are already live won’t benefit from these features.

This implies touching several repositories and creating the commits to update existing projects with new cross-cutting concerts and platform features.

In the worst case scenario, this process has to be done manually by someone. Once a new feature is available in the underlying platform, many projects are mandated to use this feature thus their respective developers are tasked to do the required refactoring. If you have Workflow CLIs or Shared Pipeline Libraries it may only require you to update to the latest version.

There are a few tools that provide distributed refactoring:

Atomist

Atomist has the concept of editors which are pieces of code that transform code. While an oversimplification, these editors can produce pull requests to the required projects thus solving the distributed refactoring problem.

I have been working with Atomist for around three years and during this time the tool has been rewritten a couple of times. That kept me from adopting it for some projects but currently, the documentation and the tool are very powerful and complete.

Snyk

Snyk has auto fixes for security vulnerabilities in the Java ecosystem. This is a more focused type of refactoring but Snyk also manages multiple projects. The need to fix vulnerabilities is something obliged in an enterprise setting and Snyk has been my tool of choice for such requirements.

So far, I have worked in two places where I have found a sad problem that I think it’s more common than I thought: having your codebase tightly coupled to your IDE.

I remember the first time I found this, I just had my first job as a tech lead for a team of Java developers. I remember that after the first day that mostly was paperwork, they showed me the place where I would sit and my machine. After that, someone sent me an email with the Standard Operation Procedure (SOP) to set up your environment.

Now, does having a SOP for something that I think should be trivial such as setting up your development environment triggers an alarm to you? At that time, with my experience at that point, it didn’t. But if I remember correctly, that document was 9 to 10 pages long removing parts like the title, table of contents and version tracking (yeah, at that time nobody used wikis to version their documents!). After reading it from top to bottom I just shook my head in disbelief.

They were running an ancient version of Eclipse at the time. And they also were tied to another very old version of Maven. They also had instructions to set up a few plugins that they had in zip format to import directly because they were discontinued. I also remember that you had to put a lot of stuff in your maven settings file in order to be able to download the dependencies of our only project… and a long list of etc.

The second time I had this problem was pretty recent, but I dare to say ten times worst. They had scripts that created the Eclipse configuration files and they saved those files to SCM. Now, of all the problems that you can imagine being the result of this, I can tell you a few:

• Hardcoded paths: because you have to save your IDE configuration to SCM, you have to explicitly setup paths in special directories or your whole project won’t work.
• Vendor Lock-in: The project won’t work on other IDEs because the configuration files are tied to the version of Eclipse that they were using, and because the version was provided by a vendor of a JEE server, the plugins also generate specific configurations that are not compatible with other versions of Eclipse.
• Developer images of the operating system: the project won’t work at all if you use other than the specific image that they provide to developers because they need special directory structures for the project to work and special network drives setup for this.

Imagine working in such a project? I can tell you that it took me at least one week to have the applications working locally. Imagine the mental overhead that a build system like this is. From time to time I still get emails from other software developers asking how I managed to have my local environment working.

The consequences of this are that in both projects were stuck in this situation and if you wanted to upgrade, let’s say, your Java version, you were stuck to what the IDE supports. Quoting one of my coworkers, it’s a sad situation that your entire team or organization is limited to what your current IDE can do.

The problem with this kind of mistakes is that they force you to take more bad decisions. I can tell you that those developer images are the symptom of a deeply broken build process.

In my opinion, if you cannot build your entire application and run it with a single command I dare to say that you have problems.

The Underlying Problem

An IDE/Text editor is something personal, we as programmers do care about efficiency and our text editors or IDEs are personal choices that allow us to become more productive in our work. Some of us pick one because we see it as a bragging right.

To the point of this article, in both these projects, someone decided that developers should use the same IDE all the time and dictated what they should do every step of the build process. This may look a logical choice for many reasons such as consistency but the underlying problem is about trust. If you do not trust your programmers to take trivial decisions such as use their own IDE I would dare to say that your organization has problems beyond these technical details.

Both projects had technical leaders that came to the wrong conclusions (that everyone should use the same IDE) with the proper reasoning (that consistency is good). The problem here is that their process focuses on preventing errors because that sounds so good when you think about it. But this kind of process “cripples innovation because it tries to prevent small mistakes” and it becomes a burden that does not prevent or even generates big mistakes (such as having a complex build process).