key terms
heap: a sampling of memory allocations of live objects, monitor current memory usage
basically this is where OS stores the objects that are used by the program, it later gets garbage collected
pprof: there is a package in go called pprof, which is used to profile the usage of the program also the tool pprof is used to analyze the memory usage of the program
memory profiling steps
import the pprof package
import _ "net/http/pprof"
starting the pprof server
go func() {
http.ListenAndServe("localhost:6060", nil)
slog.Debug("pprof server started")
}()
This step is to start the pprof server, which will be used to get the memory profiling data. Added this to the beginning of the main function.
taking a snapshot of usage
The pprof package provides a way to take a snapshot of the memory usage of the program. This can be done by sending a GET request to the /debug/pprof/heap endpoint of the pprof server. The snapshot can be saved to a file by redirecting the output of the request to a file.
curl http://localhost:6060/debug/pprof/heap > heap.out
→ Considering this, I created this script to take snapshots of the memory usage of the program. This script will take 5 snapshots of the memory usage of the program at an interval of 30 seconds.
#!/bin/bash
# Define the number of snapshots and the interval (in seconds)
num_snapshots=5
interval=30
for ((i=1; i<=num_snapshots; i++))
do
timestamp=$(date +%Y%m%d%H%M%S)
curl http://localhost:6060/debug/pprof/heap > "heap_${timestamp}.out"
echo "Snapshot ${i} taken at ${timestamp}"
sleep $interval
done
→ Don't forget to make the script executable by running chmod +x script.sh
analyzing the snapshots
The snapshots can be analyzed using the pprof tool. The pprof tool can be used to generate a graph of the memory usage of the program.
go tool pprof heap.out
This command will open the pprof tool in interactive mode. The pprof tool provides a number of commands that can be used to analyze the memory usage of the program. I used the top command to get a list of the top memory-consuming functions in the program. For my scenario, it was google cloud upload function that was consuming the most memory, was a real pain in the.. soul\
Also tried web command to get a graph of the memory usage of the program. It provided a visual representation of the memory usage of the program. This was a bit hard to understand though, but provided a vision of what's going on
I also used all the .out files generated by my script, got a good look at the memory usage of the program in time.