PC: Photo by Franck on Unsplash

The requirement:
Working on a project migrating an enterprise system to use Apache Kafka, a recent requirement was to ingest data coming in as XML documents with no pre-defined schema and convert them into the JSON format before they are published to Kafka topics.

This article focuses on how to create a custom source/sink connector using Apache Kafka and not on the actual XML-to-JSON conversion. We go through the steps we took to be able to write, build and run custom code with its dependent libraries

The tool:
Quoting Apache Kafka documentation,

Kafka Connect is a tool for scalably and…

Can order be maintained between partitions?

I recently started working on a client engagement using Apache Kafka. This is a very well established client, now in the phase of transitioning to use Apache Kafka.

In short, Apache Kafka allows the decoupling of source systems from target systems. This is especially useful when there are multiple source and target systems that all need to exchange data with each other. With Kafka, the source systems can write to Kafka, from where the target systems consume. Kafka is distributed, provides horizontal scaling and fault tolerance.

Use cases:
User activity tracking and recommendation systems, operational metrics management, and notification systems…

describe( 'CRUDService', () => {
let service: CRUDService;
let httpTestingController: HttpTestingController;
* Using sample data to check implementation of CRUD methods
const mockData = {
"status": "success",
"data": [{
"firstname": "Jane",
"lastname: "Doe",
"id": 34,
"dob": "11-02-2002"
"firstname": "John",
"lastname: "Doe",
"id": 2,
"dob": "22-09-2000"
beforeEach(() => {
providers: [
{ provide: 'url',
useValue: 'apiurl'
imports: {HttpClientTestingModule]
service = TestBed.inject(CRUDService);
httpTestingController = TestBed.get(HttpTestingController);
afterEach(() => {
it('getAll should make a GET HTTP request and return all data items', () => {
service.getAll().subscribe(res => {

Additional Notes on increasing resiliency with Delayed Retrying, Error Handling

Create a CRUDService.ts file with the Angular CLI
Modify the constructor generated by the CLI as follows. Add functions for GET, POST, PUT, DELETE HTTP requests, with custom HTTP Headers as required.

import { HttpClient, HttpHeaders } from '@angular/common/http';let HTTP_OPTIONS;@Injectable()
export class CRUDService {
* @param url URL string passed from service class that extends
* CRUDService.ts
* @param http HTTPClient service instance that enables making HTTP
* calls
constructor(@Inject('url') private url: string, private http: HttpClient) {
headers: new HttpHeaders(
{ "HeaderNameA": "HeaderValueA",
"HeaderNameB": "HeaderValueB"
* Fetches all records
getAll() {

  1. Use the Angular CLI for file generation:
    This creates the component files in a folder with the component-name as <component-name>.ts|html|css|spec.ts, making the file structure easy to read.
  2. Maintain a flat folder structure without excessive nesting of subfolders.
  3. Maintain small function definitions (Angular suggests <75 lines) whenever possible for ease of testing, reuse, and maintainability.
  4. Consistent naming conventions:
    a) Upper camel case for class names and feature.type.ts (type: component, service, module, etc.)
    b) Folder naming as <feature-name>
  5. Keep third party modules in a separate module file: this helps avoid clutter in app.module.ts …

1.Testing a DialogService that opens a Material Dialog pop-up component CommonDialogComponent with mocks and spyOn

import { DialogService } from './dialog.service';
import { CommonDialogComponent } from './src/app/components/common-dialog/common-dialog.component';
import { By } from '@angular/platform-browser';
describe('DialogService', () => {
let component: CommonDialogComponent;
let fixture: ComponentFixture<CommonDialogComponent>;
const dialogMock = {
close: () => {}
let service: DialogService; beforeEach(() => {
imports: [MatDialogModule, BrowserAnimationsModule],
declarations: [CommonDialogComponent],
providers: [
{ provide: MatDialogRef, useValue: dialogMock },
{ provide: MAT_DIALOG_DATA, useValue: [] }
it('should be created', () => {
it('should show dialog', () => {
let dialogServ = TestBed.get(MatDialog)…

The rain had caught her quite by surprise.

She liked being prepared for everything. She liked being in control. In knowing beforehand, in possibly predicting. In the safety of the thought, that she was doing the most she could, towards self-preservation, always.

‘You didn’t check the weather forecast!’, her inner voice spoke. Sharp, taunting. ‘Well, with the bright sunshine, blue skies, and white clouds, who would have thought!’, another inner voice replied, this time a softer tone, defensive. …

AngularJS vs Angular

  • Angular applications can be up to five times faster than AngularJS applications
  • Angular uses Typescript (a superset of Javascript), which with static typing points out errors during compile-time and supports modules and interfaces, not just improving performance, but avoiding many runtime pitfalls which were making AngularJS difficult to use for larger and complex applications
  • Angular comes with its own CLI for generating components, services and hence aids faster development. Routing is easier to develop with Angular
  • Angular is more intuitive
  • Angular is mobile-oriented

React vs Angular

  • React is faster both on mobile as well as for web apps (as its size is smaller…

So, he was there standing in the kitchen when I moved into my rented apartment. Seeing me, he ran inside. After the long flight from India, I was too tired. I thought, ‘Let it be, tomorrow I’m going to ask him to leave’. The day after, I searched for him with a jet-lagged pair of eyes and didn’t find him. My jet-lagged brain thought, ‘Good riddance! At least he was courteous enough to leave without being asked to!’

The next time I saw him was when I’d gotten up at night for a quick snack. He stood there for a…

Jeni Joe

Rose-Colored Glasses

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store