Weekly Bullet #35 – Summary for the week

Here are a bunch of Technical / Non-Technical topics that I came across recently and found them very resourceful.

Technical :

  • It is December and Advent of code is here. What is Advent of code ? – link here. An old podcast on Spotify’s Engineering team geeking out every December on AOC – link here
  • [A talk – 31mins ] – Concurrency is not Parallelism – “Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once.”YouTube
  • Observability is all about Metrics, Events, Logs and Traces [MELT]. The four core metrics types explained in detail here
  • From P99 Performance conference for 2022 – the power of eBPF for performance insights — link here
  • [Book]: Opentelemetry is the second most active CNCF project, next to K8. Opentelemetry is the next standard for implementing vendor agnostic observability into any application. Below is a great report on the same.

Non-Technical :

  • The perks of High Documentation, Low Meetings work culture – link here
  • Richard Feynman’s way of taking pressure off yourself and doing something for the fun of it. – link here
  • [Documentary – 40mins] – The speed cubers on netflix – link here
  • Which books have made you a better thinker and problem solver? – some great recommendations here
  • An extract from a book:

How we tend to view the worst events in History? We tend to assume that the worst that has happened is the worst that can happen, and then prepare for that.
We forget that “the worst” smashed a previous understanding of what was the worst. Therefore, we need to prepare more for the extremes allowable by physics rather that what has happened until now.

The Great Mental models, Shane Parrish

Cheers, until next time!

[Memory-metrics]: Linux /proc interface

This writeup is more of a demo to showcase the power of “proc” (process information pseudo-filesystem) interface in linux to get the memory details of process, and also a quick brief on the power of “proc interface”.

In the current trend of building abstraction over abstractions in software/tooling, very few tend to care about the source of truth of a metrics. There are various APM / Monitoring tools to get the memory details of a process for a linux system, but when the need arises, I believe, one must know the ways of going closer to the source of truth on a linux system and verify things.


So, What is proc ?

proc stands for “process information pseudo-filesystem.”
proc is a pseudo-filesystem that provides an interface to the Linux kernel. It is generally mounted at /proc and is mounted automatically by the system.

listing mounts

As seen above, on listing the mounts, you can see the device proc, which is not really a device, but is just the listed as word proc meaning that it’s a kernel interface.

In proc, you can see three different categories of information mainly.

  • sys directory – has details for files, kernel, networking, virtual memory etc of the system.
  • pid directories – which contains details of what a process is doing at process level.
  • status files like – cpuinfo, meminfo etc of the system.
a look into /proc path. pids, sys highlighted. Rest are status files.

A lot of the linux toolings like ps get the process level information from /proc path. An overview of proc at the man page – here

With that general overview of proc interface, let move to getting memory information for a process in detail for our use case.


Scenario: A java process is crash periodically and consistently. How to get the memory details for the pid using /proc interface ?

To begin with,  there are more than one ways of doing this analysis. For example: set up auto dump generators on heap(JVM params) and set up the core dump generation on ulimit. Get the dumps on crash and work backwards by analyzing them.

Since the intention here is to discover the capabilities of /prod/pid/* tooling, we will try and collect metrics from these toolings for our analysis.

First, lets collect the metrics for the java process running on the system from the /proc directory as the java process is running, so that we can analyze it. A tiny shell script for that below.

ps = "java";
seconds = "60";
dir = "memstats";
while sleep
  $seconds;
do
  ts = "$(date +'%F-%T' | tr : -)";
echo "Collecting memory statistics for process(es) '$ps' at timestamp '$ts'";
for pid
  in $ (pidof $ps);
do
  echo "- Process ID: $pid";
pdir = "$dir/$ps/$ts/$pid";
mkdir - p $pdir;
cp / proc / $pid /
{
maps, numa_maps, smaps, smaps_rollup, status}
$pdir /;
done;
done

The above script:
– creates the directory structure
– monitors the running java processes every 60secs
– copies the /proc/$pid metrics dump to the above directory structure

Let the above script run and collect the metrics, as the java process we are monitoring is getting to crash. Once we have the metrics collected, lets look in to the memory details of the pid crashing.

metrics collected before the process crashed from above script
  • The system under test had 32 GB memory in my case.
  • If we look at the vmRSS memory for the process dump, we see that java process is consuming all 32GB of the memory. Notice that status file is looked in to from /proc/pid which has the memory usage details for the process.
  • This is reflected closely by the sum of Rss values of each VM area/section collected in above dump. Note that we are digging in to smaps from /proc/pid to get these details on each VM section here for cross validation.
  • One observations on object sizes is, the VMAs with RSS on 10 MB or more (5 or more digits for kB field) are 429, which we get by looking in to smaps for the pid.
  • Before the next observation, look at one of the entires of an object in smaps file.
details of one of the objects in smaps file. Similar details for each object will be present in the smaps file.
  • smaps file for the pid in /proc has a lot of details about the residing objects which are consuming the memory. Going back to the objects of size more than 10MB, all 429 objects don’t have any file reference which were holding the memory in my case, and the allocation was Anonymous. (refer to Anonymous row in the above image)
we are trying to get all the objects which are over 10MB, and have a file reference to them. We get zero such files.
  • “Anonymous” shows the amount of memory that does not belong to any file. More details on Anonymous reference on the kernel documentation here
  • In short, what the above data points infer is, for the java process which is crashing, all the size does not come from jvm heap but comes from non-java / C-style allocation. Most probably the crash is happening JNI layer.
  • In such cases, you will not even see any heap dump getting generated. However, core dumps will be generated as diagnostic dumps for analysis if the “core file size” is set to unlimited in “ulimit“. More details on how to get core dumps here .
  • With the above details, this might be due an internal library which is used in the application which is non-java and is causing the crash.
  • From here you can look at “maps” file under “/proc/$pid/” to look at all the non ".jar” files to look at the non-java references and analyze it further.
  • In my case, it was a bunch of non-standard libraries which were packaged, that was causing the crash in JNI layer. Updating which solved the issue.

Conclusion:

  • There are always more than one ways of solving the problem.
  • The purpose of this write up again, was to show the power of understanding linux process diagnostics that come with built in with "/proc” interface.
  • Building abstractions and not letting ssh access to vms (which is the current industry trend) is great, but going closer to the source of truth can help solve problem sometimes when you know what you are looking for.

Until next time, happy tuning!

Weekly Bullet #34 – Summary for the week

Here are a bunch of Technical / Non-Technical topics that I came across recently and found them very resourceful.

Technical :

  • [Book-Recommendations]: Cloud-Native Observability with OpenTelemetry by Alex and Charity. (Side note: Half way through the book and learning a lot. )
Take away : Effective ways of adding metrics/traces in cloud native apps without marrying to any APM tools
  • A list of possible developer questions to consider asking a prospective employer. – github link
  • [Video]: Disk latency being directly proportional to vibrations (in other words, lets try shouting in data center!!) — 2min Video link
  • What is Redis explained – link here
  • Best Practices for Fixing your alerts in your service? (so that you can sleep well?) — link from newrelic here
  • The disproportionate influence of early tech decisions (debatable but interesting!) – link here

Non-Technical :

  • Second-order thinking: How to NOT create new problems while solving existing problems. – Link
  • [90mins] Podcast recommendation — The Knowledge Project : Insights for making better DecisionSpotify link
  • What’s the strangest thing you ever found in a book ? (heartwarming!)– link
  • 10 mental concepts for thinking better – Twitter thread
  • A quote that struck the right chords:

“So much advantage in life comes from being willing to look foolish in the short term.”

Unknown

Cheers, until next time!

Docker: A list of most frequently used commands

This writeup is a dump of my study notes on most frequently used docker commands for reference. This is just a self reference page and will get updated on the go

  • To run a container from an image
docker run <image name>
  • To run a docker image with a specific tag. Example below of pulling redis image with tag4.0. You will get these tag details on the dockerhub page for the image
docker run redis:4.0
  • To run a docker image in detached mode
docker run -d <image_name>
  • To run a docker image and login to the container directly
docker run -it <image_name>
  • To list all the docker images
docker images
  • To pull a docker image from dockerhub but not run it.
docker pull <image_name>
  • To list all the docker containers
docker ps -a
  • To stop a docker container
docker stop <container_name>
  • To remove a docker container form the disk.
    Note: This will remove the container permanently. It will not list anymore in docker ps -a. However, the image still exists. The exited/stopped containers do not consume any CPU or memory, but they still use the machine’s disk space.
docker rm <container_name>
  • To remove a docker image
docker rmi image
  • To execute a command in a running docker container
docker exec <container_name> <command>
  • To get the ip of a docker container
docker inspect <container_id/container_name> | grep IPAddress
  • To map the internal port of a docker container to a host port
docker run -p 80:5000 <image_name>
  • To get the logs of a container
docker logs <container_name>
  • To build a docker file
docker build . #from being in the dir which has Dockerfile
  • To map an external directory at the bootup to a docker container
docker run -v /myCustomdir:/defaultDir -u root imageName

Weekly Bullet #33 – Summary for the week

Here are a bunch of Technical / Non-Technical topics that I came across recently and found them very resourceful.

Technical :

  • A few interesting and hidden features of Python – link
  • Below are the BCP tools that can be used for digging in to Performance analysis of memory parameters on a Linux machine. More in BPF Performance Tools book by Brendan Gregg.
Book : BPF Performance Tools
  • How does Database indexing work? – here
  • How to get the most out of your 1:1s – here
  • At around 8-10years of experience, career branches in to either Engineering management or Technical Staff Engineer. Here are a few stories of Staff Engineers and their journey.
  • [Self-endorsement]: A tiny 50lines of code tool for getting my highlighted quotes from my Fav books – here

Non-Technical :

  • [Highly recommended]: First-principles thinking is a competitive advantage because almost no one does it. More here
  • The importance of annual health checkup – Seema’s True story of battling cancer – here
  • I always thought about a social platform but based on Books. Booqsi is trying to do the exact same. It is still in early beta though. More details here
  • A website that removes things from images in seconds- Magic Eraser 
  • What Got You Here Won’t Get You There: Book Summary – here
  • Podcast : Hugh Jackman – on Daily routine, intuition, meditation, decision making and more. link
  • A quote that the struck right chords:

“If you are not willing to take responsibility for your situation, you cannot use your situation as an excuse either.”

Cheers, until next time!

[Tiny tool]: Book Extract Reminders

There is no better pleasure than the Joy of solving your own problems.

This write up is not to show off coding skill(there is hardly any code in this tool), but to show the ease with which anyone can build tools to solve problems these days.

Problem statement:

How to retain the most out of the books we read? Maybe receiving daily reminders with extracts from the books?

I consume books mainly in the digital format(via kindle/calibre). I have a lot of highlights in these books which I want to be periodically reminded about. I felt, if I take out 6hours in reading a book, and completely forget all learnings in next 6months, then that’s not efficient.

Solution:

So the idea was, build something that:

  • takes all my highlights from the kindle/calibre (currently manual – to be automated)
  • pull these highlights in to git
  • python tool, that randomly pick 10(configurable) highlights
  • mails them to gmail using smtplib.
  • automate the workflow via github action to run daily at a specified time.

The code is available for anyone to take a look at here on git. Kindly go through README file for more details.

Below is the mail format that is sent daily(7am) with the extracts from the books that I have read and highlighted.


A few things:

  • If you wish to receive these mails as well, cut the PR here, with your mail id. If you are not comfortable sharing mail-id on git, mail me(akshaydeshpande1@acm.org), I will add you in to a gitingnore file.
  • Feel free to fork the repo and run it with your quotes/highlights from your books [MIT licensed].
  • If you mainly consume books as hard copies, then you can use Google Lens to get the text out of your books and add it to git.

Open for any ideas / suggestions.

Weekly Bullet #32 – Summary for the week

Here are a bunch of Technical / Non-Technical topics that I came across recently and found them very resourceful.

Technical :

  • Memory leaks on client side – the forgotten side of web performance. Link
  • A list of helpful patterns/commands on "sed" command – link
  • A “Streaming availability” api to lookup which show/movie is available in which OTT in 60+countries. Something to explore for a fun weekend project – link
  • You remember the times, when there are 10 terminal sessions opened on your system and you wanted to know “at what time did I run that command” ? Ya, article addresses the same problem – link
    PS: If you use zsh, there a few themes which come in-built with this feature.
  • Wordle puzzles are crazy popular in the last week or so. Here is a python project to solve the puzzles. link

Non-Technical :

  • Completing part-time master’s in CS while on a full time job! This article was so inspiring also reminds, how much time we waste in general. – link
  • Drop a raindrop anywhere in the world and watch where it ends up. A fun site – link
  • Rocket engines, they deal with a lot of heat right? How come they don’t melt? Detailed geeky explanation – here
  • An interesting Thread on Human psychology fact. link
  • [Image below]: “Stop focusing on the black lines behind you. Start focusing on all of the green lines before you.”
SOURCE: Twitter
  • Extract from a book:

“When you worry, ask yourself, ‘What am I choosing to not see right now?’ What important things are you missing because you chose worry over introspection, alertness or wisdom?”

The Obstacle in the Way, Ryan Holiday

Have a great week ahead!

Weekly Bullet #31 – Summary for the week

Here are a bunch of Technical / Non-Technical topics that I came across recently and found them very resourceful.

Technical :

Source: Unknown

Non-Technical :

  • Naval Ravikant is a great thinker who makes you wonder on different thought processes. His reading recommendations here
  • Lessons from my PhD” – Not only on the PhD topic but in general by Prof. Austin.
  • If— a wonderful poem by Rudyard Kipling here
  • Although, this article is not very engaging in anyways, but the simplicity of it made me fall in love with cycling again. — “Cycling to work
  • We all know the Nobel laureate Richard Feynman for his unique way of thinking(example) but here are some Heart-Wrenching Letters from him to his late wife, Arline.
  • Unless you were living under a rock, you know about James Webb Telescope launched by NASA on Dec 25th. It is fully deployed now. Here is a great comment explaining why it is a Big Deal. A video here.
  • Extract from a book:

“When you are not practicing, remember, someone somewhere is practicing. And when you meet them, they will win.”

Ed Macauley

Happy and Peaceful 2022 !

Weekly Bullet #30 – Summary for the week

Here are a bunch of Technical / Non-Technical topics that I came across recently and found them very resourceful.

Technical :

  • P99 CONF (centered around low-latency, high-performance design) recordings are available here

Non-Technical :

  • Traveling(rather walking) without money! Although old(1998) event, still shows world is not that bad of a place. Link “MY PENNILESS JOURNEY
  • There was this mega-thread on twitter for “one book that changed the way you see the world”. – The consolidated list from the thread on BooksChatter here.
  • Matthew McConaughey addressing University of Houston outgoing students — “5 Rules for the life” (Rule #1 is my fav)
  • Extract from a book:

“If you think about the biographies you read or the documentaries you watch about the greats in various fields, this same pattern of Addictive, Passionate behavior surfaces. Jazz saxophone great John Coltrane reportedly practiced so much that his lips would bleed.”

The Passionate Programmer

Have a great week!

Weekly Bullet #29 – Summary for the week

Here are a bunch of Technical / Non-Technical topics that I came across recently and found them very resourceful.

Technical :

  • eBPF Summit is live now – Recording of the Keynote and live summit here
  • Conference talk – USENIX LISA2021 Computing Performance: On the Horizon by Brendan Gregg – here
  • A project for visualizing codebase – here
  • Self healing systems – the real end goal of observability – here
  • Python3 – Reverse Engineering Tips – here

Non-Technical :

  • Great article on – “How to think: The skill you’ve never been taught” – here
  • An addictive trading game in your browser – Paper trade – here
  • People no longer trust each other. Why? And how can we fix it? An interactive guide to the game theory of trust – here
  • What tiny purchases have disproportionately improved your life? – Thread link
  • Extract from a book:

“But we had two choices,” I said. “Throw our hands up in frustration and do nothing, or figure out how to most effectively operate within the constraints required of us. We chose the latter.

Extreme Ownership by Jocko Willink;Leif Babin