Where It All Started.

Where It All Started.

Life, Stock Trading, Investments, Business and Startup. Most are programming stuff.

Category: DevOps

Limit Window Subsystem Linux v2 (WSL2) Resources To Speed Up Kubernetes

Window Subsystem Linux v2 (WSL2) is an iteration of the VM created by Microsoft, from Hyper-V to WSL and this the second generation of WSL. If it’s your first time accessing WSL2, it automatically provide you with the default setup which doesn’t provide any limits accessing your full workstation resources (CPU, RAM and other HDD). It means that if you have 8 cores cpu and 16Gb memory, it will use all that up. The problem with it is sometimes it affects your host computer and it gets slow. So to solve that problem we try to limit the resource consumption of WSL2.

Photo by Sadik Brika on Unsplash

Limit WSL Resource Consumption

On your profile directory %USERPROFILE% create a new file named .wslconfig. Set it’s content to the following:

[wsl2]
memory=8GB
processors=8

Change the settings base on your workstation capability, and this is what works for me.

Next, open up a powershell terminal in administrator mode and restart the LxssManager as this manages WSL2.

Get-Service LxssManager | Restart-Service

You could also use the wsl --shutdown method to restart WSL. Check if the vmmem process still consumes beyond its limit.

Troubleshoot

If the changes still not reflecting, try to restart your machine and also restart Docker Desktop.

Move Docker Desktop Data to Another Location (WSL 2)

In Docker Desktop for Windows the WSL2 version, you don’t usually have options to increase memory and diskspace as it will be managed directly by Windows.


The Docker Desktop data can be found originally in this location %USERPROFILE%\AppData\Local\Docker\wsl\data.

🚚 Export Docker Data

In order to make this work, first shutdown Docker Desktop. This can be done by right-clicking the system tray icon of Docker then from the context menu Quit Docker Destop.

Next is open your command prompt and type the following:

wsl --list -v

On which, when run will return to you the state of all WSL images.

  NAME                   STATE           VERSION
* docker-desktop         Stopped         2
  docker-desktop-data    Stopped         2

After that we export the docker-desktop-data into a tar archive. We will assume you are planning to move the docker data into D: drive, and within the drive you have already created a folder named Docker.

wsl --export docker-desktop-data "D:\docker-desktop-data.tar"

Next, is to unregister docker-desktop-data from WSL.
This command below will delete ext4.vhdx from %USERPROFILE%\AppData\Local\Docker\wsl\data\ext4.vhdx, so make sure you back it up first.

wsl --unregister docker-desktop-data

🚛 Import Docker Data

After export, we do import docker-desktop-data back to WSL.

wsl --import docker-desktop-data "D:\Docker" "D:\docker-desktop-data.tar" --version 2

The ext4.vhdx will now reside in the D:\Docker folder. Start Docker Desktop and verify the changes.

If everything works out, you can now delete the tar archive you created earlier D:\docker-desktop-data.tar. Please don’t delete the ext4.vhdx, otherwise you would lose all your images and containers in docker.

In case docker icon turns red in Docker Desktop, clear the docker cache which can be found in Docker Desktop settings.

Tunnel In Existing SSH Connection

Remote work, is a blessing and sometimes nightmare depending in your line of work. I’ve been in a situation where I’m connected to a remote workstation but due to some technicalities I’m not allowed to disconnect the current SSH1 connection and or create a new one. And where it lies, I need to tunnel a service from the remote workstation to my local machine.

So here’s how I did it!

The real voyage of discovery consists not in seeking new landscapes, but in having new eyes.

— Marcel Proust.

So where do we start?

Once you have an existing SSH session opened using the default OpenSSH2 client, to open a tunnel simply type <enter>~C where <enter> is the key on your computer keyboard.

~ (tilde) is the SSH’s default EscapeChar. You press <enter> first to clear the buffer, the ~ escape char and any one of a number of options.

If all goes well it will bring up a new console associated with your local SSH client, that will accept SSH command flags, which includes -R and -L.

To map a server service to your local workstation you need to use -L flag. The arguments for that flag would be [bind_address:]port:host:hostport but normally the bind_address is optional.

Then if you want to map local service and tunnel it to remote server, you’ll need to use -R flag. This flag holds similar arguments to the -L.

For example, if I want to forward a remote server Nginx deployed website and access it locally (with local bind IP). What could I do is type <enter>~C then -L 80:localhost:8080<enter>, after that I will immediately gain access to that when I access the site using localhost:8080 on my local machine.

To get a full list of escape sequence that the OpenSSH client accepts, type <enter>~?:

Supported escape sequences:
 ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~C   - open a command line
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice

That’s all guys. 🐲

Conclusion

Most of the command line tools have flags you probably haven’t explored. So try to explore each one to become proficient in the platform you are currently working on. Just like programming, you won’t memorize it on a day, but to truly know the tools capability you must use it in a very dire situation.

This OpenSSH2 escape sequence is really helpful for DevOps and software engineers (for software development).

Let me know in the comments if you have questions or queries, you can also DM me directly.

Follow me for similar article, tips, and tricks ❤.


  1. SSH or Secure Shell is a cryptographic network protocol for operating network services securely over an unsecured network. ↩︎
  2. OpenSSH (also known as OpenBSD Secure Shell) is a suite of secure networking utilities based on the Secure Shell (SSH) protocol, which provides a secure channel over an unsecured network in a client–server architecture. ↩︎

Powershell Symbolic Links in Windows 10

Recently, I’ve been using more the Powershell1 prompt rather than the old command prompt2. Both command consoles can still be run on Windows 10, but on recent occasion I prefer the Powershell as you can use it to create more complex shell scripts on Windows and access some C# modules.


A chain is only as strong as its weakest link.

— Anonymous.

On my previous recent post about moving ProgramData to another drive, I’ve use the mklink utility to create junction directory to-and-from. So here are the equivalent commands:

Command Prompt SyntaxPowershell Equivalent Syntax
mklink Link TargetNew-Item -ItemType SymbolicLink -Name Link -Target Target
mklink /D Link TargetNew-Item -ItemType SymbolicLink -Name Link -Target Target
mklink /H Link TargetNew-Item -ItemType HardLink -Name Link -Target Target
mklink /J Link TargetNew-Item -ItemType Junction -Name Link -Target Target

The New-Item command is also analogous to Unix touch command tool.
Check the definition first of those commands before running on your system.
That’s all guys!

Leave a comment if you have questions and queries. Also you can DM me on twitter 😉.

💻


  1. PowerShell is a task automation and configuration management framework from Microsoft, consisting of a command-line shell and the associated scripting language. Initially a Windows component only, known as Windows PowerShell, it was made open-source and cross-platform on 18 August 2016 with the introduction of PowerShell Core. ↩︎
  2. The name refers to its executable filename. It is also commonly referred to as cmd or the Command Prompt, referring to the default window title on Windows. The implementations differ on the various systems but the behavior and basic set of commands is generally consistent. cmd.exe is the counterpart of COMMAND.COM in DOS and Windows 9x systems, and analogous to the Unix shells used on Unix-like systems. ↩︎

Identity Server 4 On Kubernetes Nginx Ingress

Have you ever tried deploying Identity Server 41 on a k8s (Kubernetes2) setup with Nginx3 ingress?

If you tried, I’m sure you’ve encountered some problems, as the current Nginx ingress is not properly configured for ASP.Net project or does not contain better optimization for Identity Server 4.


The first step towards getting somewhere is to decide you’re not going to stay where you are.

— J.P. Morgan.

Come on join me as we dive into the configurations!

Prerequisites

First of all, you must have a Kubernetes on your machine. Second, must have existing test bed project for Identity Server 4.

If you don’t have Kubernetes, perhaps you could try installing MicroK8s. The MicroK8s works on windows and MacOS.

So where do we start?

First, we modify the ingress ConfigMap configuration, and add the following lines:

proxy-buffer-size: "128k"  
proxy-buffers: "4 256k"  
proxy-busy-buffers-size: "256k"  
client-header-buffer-size: "64k"  
http2-max-field-size: "16k"  
http2-max-header-size: "128k"  
large-client-header-buffers: "8 64k"

This specific modifications allows Identity Server 4 to send and receive large header data which is needed to store and sort out JWT (JSON Web Token) identifiers. You can check this sample setup on my test ingress config map YAML (Yet Another Markup Language):

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-ingress-nginx-ingress
  namespace: default
  selfLink: /api/v1/namespaces/default/configmaps/nginx-ingress-nginx-ingress
  uid: 9fe8c06b-4f7c-4032-a938-505c308ed332
  resourceVersion: '10291469'
  creationTimestamp: '2020-09-18T12:46:50Z'
  labels:
    app.kubernetes.io/instance: nginx-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx-ingress-nginx-ingress
    helm.sh/chart: nginx-ingress-0.6.1
  annotations:
    meta.helm.sh/release-name: nginx-ingress
    meta.helm.sh/release-namespace: default
data:
  client-header-buffer-size: 64k
  http2-max-field-size: 16k
  http2-max-header-size: 128k
  keepalive-timeout: '65'
  large-client-header-buffers: 8 64k
  proxy-buffer-size: 128k
  proxy-buffers: 4 256k
  proxy-busy-buffers-size: 256k
  proxy-http-version: '1.1'
  proxy-read-timeout: '150'
  sendfile: 'on'
  use-http2: 'false'

Next, thing we do is adjust our code to forward headers from and to ingress-app. The other method calls are also recommended by docs from Microsoft, you can check the setup here.

public void ConfigureServices(IServiceCollection services)  
{
    // ... code omitted ...
    // Needed for load balancer to forward headers
    services.Configure<ForwardedHeadersOptions>(options =>
    {
        options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto;
        options.RequireHeaderSymmetry = false;
        options.KnownNetworks.Clear();
        options.KnownProxies.Clear();
});

The docs specified the known networks / proxies are needed if you are hosting C# apps in non-windows hosting environment.

After adding a forward headers configuration onto our ConfigureService method. We also need to add the forward headers middleware on the Configure method, can also be found in Startup.cs file.

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    // ... code omitted ...
    app.UseForwardedHeaders();
    // ... code omitted ...
}

Then after that, restart the Nginx ingress and also your app to test whether everything is working fine. The next change is optional if you are using TLS.

If your ingress setup is TLS4 terminated. You also need to add this on your Configure method.

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    // ... code omitted ...
    app.Use(async (ctx, next) =>
    {
       ctx.Request.Scheme = "https";
       await next();
    });
    // ... code omitted ...
}

This specific custom middleware specifically converts all incoming calls to secured HTTP scheme. The TLS ingress specifically does is redirect the calls from your RS (Resource Server) to AS (Authorization Server) which is Identity Server 4 but TLS needs consistent HTTP secured scheme. If you look into your openid-configuration it will return http:// only endpoints and that is the problem, and that’s why we are modifying it internally using a custom middleware.

After all is done, restart the service and test every knick and knacks.
That’s all guys!

Conclusion

It’s not just a simple clone image and deploy setup in k8s especially if you’re trying to deploy a c# app, sometimes you need to optimize some config in order for it to run smoothly /and or work well. Check the recommended deployment guide in Microsoft docs.

Let me know in the comments if you have questions or queries, you can also DM me directly.

Follow me for similar article, tips, and tricks ❤.


  1. IdentityServer is an OpenID Connect provider – it implements the OpenID Connect and OAuth 2.0 protocols. ↩︎
  2. Kubernetes is an open-source containerorchestration system for automating computer application deployment, scaling, and management. ↩︎
  3. Nginx (pronounced “engine X”, /ˌɛndʒɪnˈɛks/ EN-jin-EKS), stylized as NGINX, nginx or NginX, is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. ↩︎
  4. Transport Layer Security (TLS), and its now-deprecated predecessor, Secure Sockets Layer (SSL), are cryptographic protocols designed to provide communications security over a computer network. Several versions of the protocols are widely used in applications such as web browsing, email, instant messaging, and voice over IP (VoIP). Websites can use TLS to secure all communications between their servers and web browsers. ↩︎

Moving ProgramData Folders To Other Drive Using Windows 10

My C: drive became full, and it came to my mind that its hard to move files to a new SSD1 if I buy new one.

So it got me into thinking what are the things I can do to remove and free up space in my C: drive?


I don’t know where I’m going from here, but I promise it won’t be boring.

— David Bowie.

The first thing that comes up, is using the tool Disk Cleanup bundled with Windows 10. It only freed up 10Gb of data, then I check all the folder size which contains the largest amount of data.

The result was my user account and the ProgramData folder.
Here are the things I did in order to move ProgramData contents to my other spare drive.

DISCLAIMER: Before doing this on your machine please test and research first each command before executing on your machine / production environment.

First, I copied and mirrored the ProgramData folder structure and ACL’s2 using the command robocopy. The /MIR flag tells robocopy to retain security settings and state of file.

robocopy /XJ /MIR "C:\ProgramData" "D:\ProgramData"

You could also use this other command flags, this command is non-destructive unlike the mirror flag. The mirror flag deletes the file at destination while this just overwrites and retain if missing in source.

robocopy /xj /s /copyall C:\ProgramData D:\ProgramData

After everything’s done copying, you start creating junction links and symlinks3 from your spare drive (for me its the D: drive). The %~NA tells the batch command it will only get the base folder name, and the %~A gets the whole absolute path. The command below will only create directory junctions to begin with:

FOR /D %A IN ("D:\ProgramData\*") DO (MKLINK /J "C:\ProgramData\%~NA" "%~A")

This next command, specifically create symbolic links to file from source to destination.

FOR %A IN ("D:\ProgramData\*") DO (MKLINK "C:\ProgramData\%~NXA" "%~A")

Then after that restart your machine, and ensure everything’s working fine. I think some folders like Microsoft and Packages should be excluded in copying and making junctions.

That’s all guys. If you have any question DM me or comment in this post.


  1. A solid-state drive (SSD) is a solid-state storage device that uses integrated circuit assemblies to store data persistently, typically using flash memory, and functioning as secondary storage in the hierarchy of computer storage. It is also sometimes called a solid-state device or a solid-state disk, even though SSDs lack the physical spinning disks and movable read–write heads used in hard disk drives (HDDs) and floppy disks. ↩︎
  2. An access-control list (ACL) is a list of permissions associated with a system resource (object). An ACL specifies which users or system processes are granted access to objects, as well as what operations are allowed on given objects. ↩︎
  3. A symbolic link (also symlink or soft link) is a term for any file that contains a reference to another file or directory in the form of an absolute or relative path and that affects pathname resolution. ↩︎

Top 10 NMAP Flags That I Use Daily

It is not the monsters we should be afraid of; it is the people that don’t recognize the same monsters inside of themselves.

— Shannon L. Alder.

If you’re a network IT (Information Technology) engineer or cybersecurity professional for sure you’d know about the tool nmap.

The tool nmap which stands for network mapper 1 is an open source tool for network discovery and is mostly use for security auditing. Been using this tool for many years and this are my favorite command line flags:

Skip reverse DNS call

This is a helpful flag specially if you don’t want that additional millisecond of fetching records from a DNS server. Or you have a specific case scenario that involves using only internal cached host file.

nmap -n scanme.nmap.org

Stop ping checks

The -PN flag specifically tells nmap that the host is online, skipping check if its alive through ping2. This is particularly useful in situation where you know the target is blocking all ICMP (Internet Control Message Protocol)3 in firewall.

nmap -PN scanme.nmap.org

Fingerprint scan

This -sV flag is useful specially in network auditing and determining if there are any ports available. The command will probe the target machine ports availability and guess the service (including the service version) that is running.

nmap -sV scanme.nmap.org

Finding live host

This command is specifically useful for network engineers to know if there are any alive host on the network. The notation below tells to scan the specific subnet4 using ICMP protocol and return the list of host that responded.

nmap -sP 192.168.1.1/24

Scan using specified network interface

If you have multiple NIC’s (Network Interface Controller)5 and you want to route the scan to a specific NIC, then this is the solution. Normally nmap or any other tool that utilize the computer network would use the OS designated network route (normally determined by network table and preferred gateway). The -e flag tells nmap to use that specific network controller to perform/resolve the scan.

nmap -e eth0 scanme.nmap.org

SYN ping scans

The SYN scan specifically tries to send request packets to target machine and check if it accepts the request packets. Mostly this is one of the default alternative ways of checking if the host is alive.

nmap -sP -PS scanme.nmap.org

ACK ping scans

The ACK scan is the opposite of SYN. In which this particular scan sends and ACK or (acknowledge) packet to the target machine if it will respond. Most modern firewalls block this if its not associated in a three way handshake.

nmap -sP -PA scanme.nmap.org

UDP port scans

This UDP6 port/ping scan is helpful when you know the target machine only blocks TCP packets. This specific flag sends a UDP packet to ports available on the machine and check’s if the target machine responds.

nmap -sP -PU scanme.nmap.org

IP (Internet Protocol) ping scans

Actually, this particular scan is special as its send IP packets to the specified IP protocol number in their IP header. It’s kinda special in a sense that if you didn’t supply a protocol type it will send multi-packets ICMP, IGMP, and IP-in-IP packet.

nmap -sP -PO scanme.nmap.org

ARP ping scans

This particular scan is mostly useful in LAN scenario. As you send an ARP packet it will return specific address or addresses that consumed the broadcast request.

nmap -sP -PR scanme.nmap.org

Mostly, that’s all. I’ve used other flags but this are my most used command flags for nmap.


  1. Nmap (Network Mapper) is a free and open-source network scanner created by Gordon Lyon (also known by his pseudonym Fyodor Vaskovich). ↩︎
  2. Ping measures the round-trip time for messages sent from the originating host to a destination computer that are echoed back to the source. The name comes from active sonar terminology that sends a pulse of sound and listens for the echo to detect objects under water. ↩︎
  3. The Internet Control Message Protocol (ICMP) is a supporting protocol in the Internet protocol suite. It is used by network devices, including routers, to send error messages and operational information indicating success or failure when communicating with another IP address, for example, an error is indicated when a requested service is not available or that a host or router could not be reached. ↩︎
  4. A subnetwork or subnet is a logical subdivision of an IP network. ↩︎
  5. A network interface controller (NIC, also known as a network interface card, network adapter, LAN adapter or physical network interface, and by similar terms) is a computer hardware component that connects a computer to a computer network. ↩︎
  6. The User Datagram Protocol (UDP) is one of the core members of the Internet protocol suite. The protocol was designed by David P. Reed in 1980 and formally defined in RFC 768. With UDP, computer applications can send messages, in this case referred to as datagrams, to other hosts on an Internet Protocol (IP) network. Prior communications are not required in order to set up communication channels or data paths. ↩︎

Find Public IP in Linux Using CLI

A musician must make music, an artist must paint, a poet must write, if he is to be ultimately at peace with himself. What a man can be, he must be

— Abraham Maslow.

Hey guys, ever been in a situation you need to query your public WAN IP without any browser. Glad you came to the right post.

Prerequisites

  • dig – Dig stands for (Domain Information Groper) is a network administration command-line tool for querying Domain Name System (DNS) name servers. It is useful for verifying and troubleshooting DNS problems and also to perform DNS lookups and displays the answers that are returned from the name server that were queried.
  • curl – cURL is a computer software project providing a library and command-line tool for transferring data using various network protocols. The name stands for “Client URL”, which was first released in 1997.
  • wget – GNU Wget is a computer program that retrieves content from web servers. It is part of the GNU Project. Its name derives from World Wide Web and get. It supports downloading via HTTP, HTTPS, and FTP. Its features include recursive download, conversion of links for offline viewing of local HTML, and support for proxies.

What are the ways to find my IP?

Here are ways to find your own public IP from the terminal.

First is using OpenDNS servers. The OpenDNS servers are always free and a toolkit for a network engineer.

dig +short myip.opendns.com @resolver1.opendns.com

If resolver1 isn’t responding try the resolver2 to get your query.
Also you could achieve this query using Google’s nameservers.

dig TXT +short o-o.myaddr.l.google.com @ns1.google.com

The second is using curl or wget which is more common on many distro’s without installing any other packages. While this is much slower in resolving, the tools used is much more common.

curl https://ipinfo.io/ip

For wget command just replace the curl word above.

Conclusion

I haven’t listed all the ways, as there are many ways to achieve this. But all this commands have been tested and used by me several times, it always save my ass when there’s a problem. If you have additional command that you want to share DM me at @ffimnsr.

Reset Start Menu Layout Windows 10

Success is neither magical nor mysterious. Success is the natural consequence of consistently applying basic fundamentals.

— E. James Rohn.

Just recently on my Windows workstation a bug occurred on which the Recycle Bin doesn’t refresh its icon on desktop. I know for a fact that I’ve recently deleted some file so there should be contents inside the bin.

Let’s jump in!

What are the steps to reset explorer?

Here are the steps I’ve created for when I do not need to restart the PC to reset overall explorer settings or when its has flaw.

  1. Open the “Run command” window by pressing WIN+R.
    (i.e. hold down the Windows key and then press R.)
  2. Type cmd at the prompt, and press Enter.
  3. Wait for the command prompt to open.
    (It will be a flashing cursor block.)
  4. At the command prompt, type this: taskkill /IM explorer.exe /F You should notice these all vanish: Start Menu, Taskbar, any open File Explorer windows.
  5. Type this at the command prompt explorer.exe. Now those components should all load back in.
  6. Close the command prompt and try the Start Menu or Desktop. Hopefully it should have refreshed.

That’s it all done!

Conclusion

If there is a bug, there is always a workaround. Anyways, I’ve already reported the bug Microsoft feedback.

Follow me for similar tips and tricks at @ffimnsr.

Top 3 DNS Providers That Provides Good Service in SEA (Southeast Asia)

If you give a hacker a new toy, the first thing he’ll do is take it apart to figure out how it works.

— Jamie Zawinski.

DNS or Domain Name System sometimes called the phonebook of the internet is one way for us to easily access our favorite website, it translates and redirect human readable domain names (e.g. yahoo.com, google.com) to their respective IP address. Finding the good DNS provider is vital in accessing information that is sometimes censored by our government, ISP and the likes. This are my top three good DNS (Domain Name System) providers that are really fast, secure and reliable that can be use in Southeast Asia.

Cloudflare

Cloudflare Public DNS (IPv4)

1.1.1.1
1.0.0.1

Cloudflare Public DNS (IPv6)

2606:4700:4700::1111
2606:4700:4700::1001

So why Cloudflare1? Choose Cloudflare if you want less than <1ms of domain name resolution. Seriously, they have the fastest name resolution on the internet.

Google

Google Public DNS (IPv4)

8.8.8.8
8.8.4.4

Google Public DNS (IPv6)

2001:4860:4860::8888
2001:4860:4860::8844

So why Google2? You will use this if you need an old but still good reliable DNS server. Its been used as a primary name resolution on big companies as well as local workstation. Its much more better than your ISP (Internet Service Provider) provided DNS.

Yandex

Yandex DNS (IPv4)

77.88.8.8
77.88.8.1

Yandex DNS (IPv6)

2a02:6b8::feed:0ff
2a02:6b8:0:1::feed:0ff

So why Yandex3? You’ll choose this if you’re a webmaster due to super fast name resolution propagation. This is the fastest DNS for name resolution propagation, on which ever region you are currently.

These three are mostly corporate but they do provide fast speed TLD name resolution. And if you really still feel they can be manipulated, feel free to enable DNSSEC. For example on CloudFlare, a single domain would resolve at less than <1ms.

So guys, what are your top DNS providers?


  1. Cloudflare, Inc. is an American web-infrastructure and website-security company, providing content-delivery-network services, DDoS mitigation, Internet security, and distributed domain-name-server services. Wikipedia ↩︎
  2. Google LLC is an American multinational technology company that specializes in Internet-related services and products, which include online advertising technologies, search engine, cloud computing, software, and hardware. It is considered one of the Big Four technology companies, alongside Amazon, Apple, and Facebook. Wikipedia ↩︎
  3. Yandex is a technology company that builds intelligent products and services powered by machine learning. Our goal is to help consumers and businesses better navigate the online and offline world. Since 1997, we have delivered world-class, locally relevant search and information services. ↩︎