OpenWRT, dnsmasq, dhcp, static mappings, but No DNS Resolution

The Problem

After poking around WAYYYYY too long to connect these dots, I finally have a reasonable explanation for why dnsmasq, running on on OpenWRT, absolutely refused to resolve an IP address from a hostname that was configured, in DHCP, with an IP for a specific client mac address.  The client machine was online, and reachable by that IP.

TL/DR

dnsmasq only resolves names from DHCP active leases, not from DHCP config.  (Read on if you need to know how to check, how to fix, etc.)

The circumstances were:

• ip/name configured in DHCP for client machines mac address (as mentioned already).
• client machine hostname configured to match the name also assigned by DHCP
• client machine (linux) had been rebooted, after configuring DHCP and hostname
• command dhclient -r issued repeatedly on the client machine
• interface, domain, etc. set to "lan"
• ipv6 disabled (just in case it was making things more complicated).
• ping and nslookup from a different machine received NXDOMAIN response from dnsmasq/openwrt

Extra Circumstances (that didn't seem relevant, but were):

• client machine had unexpired, cached DHCP lease info in /var/lib/dhcp/dhclient.eth0.leases
• openwrt dhcp did NOT have an active lease registered in /tmp/dhcp.leases
• dnsmasq had been restarted, the entire openwrt router had been restarted, and several updates had been "Saved and Applied" via LuCI.  This probably meant that any DHCP in-memory name:ip mappings were gone.

What fixed it?

On the client/host machine that can't be resolved/found by name...

1. Delete /var/lib/dhcp/dhclient*.leases
2. Reboot (because dhclient -r eth0 didn't help).

Verification

1. Check that ping and nslookup now work as expected from other clients for the previously (let's hope) unresolvable hostname.
2. Check that openwrt /tmp/dhcp.leases DOES now include a line for the recently, forcibly-renewed DHCP lease.

Explanation:

openwrt runs dnsmasq with a configuration that references the /tmp/dhcp.leases file.  

• The setting is found on the Network - DHCP and DNS page in LuCI, on the Resolv and Hosts Files "tab" in the Leasefile field.
• The setting is found in the /etc/config/dhcp file in the config dnsmasq section as option leasefile '/tmp/dhcp.leases'
• See: https://openwrt.org/docs/guide-user/base-system/dhcp
• This says: "leasefile stores leases in a file so they can be picked up again if dnsmasq is restarted."

    

Gogs: Internal Error from "git push"

This might have been fixed in Gogs, and maybe it just got me because I'm just running an older version, but I think it's worth a quick post here anyway...

Problem

Pushing a git repository to a remote hosted in Gogs... with ssh config, etc. all as it should be... results in:

me@myworkstation#:~/mylocalgitclonedir $ git push

Gogs: Internal error

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

Cause

It turned out to be root:root ownership of the xorm.log file on the Gogs server.

In my case, running in a docker container, that was /app/gogs/log/xorm.log

This might be related: https://github.com/gogs/gogs/issues/4990

Don't know how this got changed in my particular setup... yet.  Maybe a backup or log rotation hiccup.

So, the cause was not: (including some of this for searchability of this article)

• repository write permissions for user
• SSH config on server or client
• i.e. not an ssh client known_hosts error
• i.e. not related to private key file permissions on the client
• i.e. not using the wrong private key for the registered public key in gogs
• checked with command:  ssh -v -T git@mygogshost

Solution

Open a terminal on the gogs host, and...

sudo chown git:git /app/gogs/log/xorm.log

After that git push worked fine again.

References

• https://github.com/gogs/gogs/issues/4990

HomeAdvisor.com Password Reset

This info is current as of Dec 5, 2023, so if you're reading this later, and that was ages ago, it's not likely to be correct still.  So, with that said...

I just had the "wonderful" experience of trying to get logged in to HomeAdvisor.com on a mobile browser after initiating an account from a desktop web browser.

Their "clever" scheme to use a temporary login link, sent to an email address, instead of a password, makes it impossible to CHANGE your password once you are logged in, because you might have NEVER SET a password.  The "Forgot Password" link just sends you the non-password login link again... which is just another trip around their endless loop.

Called customer support and described the issue, and they just hung up on me, which might be an indication of whether you'd even want to work with HomeAdvisor/Angi in the first place.

The only way I could get out of the "catch 22" they created on the HomeAdvisor.com site, was to guess that maybe all of the the accounts are wired into the same place as Angi.com.  So, if you go to Angi.com's login page:  https://www.angi.com/login , you'll find a "Forgot Password? link that still goes to another page, which asks for your email address (i.e. login id), and actually sends you a link to reset your password on angi.com.  Magically, that ALSO sets your password on HomeAdvisor.com.

Since the idiots in customer support can't be bothered to help anyone figure out the !@#$% mess they've made of the logins between HomeAdvisor.com and Angi.com, I hope this helps, at least until they fix it, or (more likely) make it into an even bigger mess.

Legacy NVIDIA Driver for Ubuntu 20.04+

Summary

Either because an upgrade wrecked a working Linux install, or because the default drivers aren't performing very well, it may be necessary to install NVIDIA drivers for very old, "legacy" hardware on Ubuntu, or a similar Linux distribution.  This post is meant to help get that done without burning as much time searching for the right information and putting all the pieces together.

Symptoms

• After an update / upgrade, Linux stalls mid-boot with a black screen and a blinking text cursor in the upper left corner.
• Linux installed on a machine with older / legacy NVIDIA hardware is running too slow to be usable, and the graphics driver is suspected as the cause.

Elements of the Problem

• Recent versions of Ubuntu install a kernel version that is no longer compatible with older NVIDIA drivers.
• The default, generic driver doesn't perform very well (or at all) with older NVIDIA graphics card hardware.

Getting to a Terminal

When/if the X-Window / GUI Startup failed and stopped the bootup, it may be necessary to switch to another tty/terminal.  There are typically several of them started with Linux, and they can be accessed using CTRL+ALT+F2 (or F3, F4, ...).  From the blank / black screen with the blinking cursor, use that key combo to switch over and directly log-in to a terminal / tty, where the commands below can be executed.

Diagnostic Steps

• Verify old NVIDIA hardware
• Command:   sudo lspci | grep VGA
• Should display information about the NVIDIA hardware (e.g. GeForce 9600M GT)
• Verify old, no longer working, NVIDIA driver version is installed - (e.g. 340.108)
• Command: dkms status
• Command: apt list --installed | grep nvidia
• Should show the NVIDIA driver package version (e.g. nvidia-340 ... 340.108)
• Verify NVIDIA driver is malfunctioning
• Command: nvidia-smi
• Should report a failure to communicate with graphics hardware.
• Check NVIDIA settings
• Command: nvidia-settings
• May report config errors, etc. and/or other useful info.

Elements of the Solution

• Add the PPA repository from user kelebek333 which contains a version of the NVIDIA drivers that has been patched to work with newer kernel versions.
• See: https://launchpad.net/~kelebek333/+archive/ubuntu/nvidia-legacy
• Command: sudo add-apt-repository ppa:kelebek333/nvidia-legacy
• Run command: sudo apt-get update
• This should report that one or more packages (which include patched NVIDIA driver) are upgradable.
• Verify w/ command: apt list --upgradable
• Run command: sudo apt upgrade nvidia-340
• This may report that there is a public key missing.  If so, run the following command replacing {key-value} with the key that is reported in the error message from apt upgrade...
• Command: sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys {key-value}
• Reboot - If this resolved the NVIDIA driver issues, the X Window system should start normally again.

Docker "exec format error"

Overview

Occasionally, for a variety of reasons, a Docker container won't start, and the only clue is a cryptic message that says something like this:

standard_init_linux.go:178: exec user process caused "exec format error"

I'm aware that there are numerous related forum posts to be found via web search with a collection of right, wrong, confusing, and sometimes tediously wordy responses.  This is just an attempt to cut through some of that and cover only the non-wild-goose-chase stuff, not to take credit for any of it.  If any of this ended up worded similarly to something else, it's purely coincidence.

Cause #1 - Missing "shebang" in Shell Script

Sometimes, this is related to an improperly authored shell script, serving as the entrypoint for the container image, and the solution is to add the required "shebang" (#!/bin/sh) line at the top of the shell script.

Troubleshooting

1. Run the image as a "shell" container and examine the entrypoint script 
   docker run --rm -it --entrypoint [] myimagename:myimagetag /bin/sh
2. (Possible) Solution: Rebuild the container image with a corrected script

Cause #2 - Image/Runtime Platform Mismatch

The other main reason for the "exec format error" is that the platform for which the container image was built does not match the platform where the container is running.

This could be related to how the image was fetched / staged.  For instance, using "docker pull" from a public registry and "docker push" to a private registry, using a Macbook with an M1 processor (arm64 platform) when the container will be run in a kubernetes system that is on amd64 (intel compatible) hardware and/or virtual machines.

• Note: When the docker image entrypoint references a binary executable (not a startup shell script... i.e. no chance of missing "#!/bin/sh"), it is more likely that the "platform mismatch" is the cause of the "exec format error".
• Note: If the source registry does not have an image matching the client/target platform (e.g. on a Raspberry Pi Arm 32bit OS), docker pull might fetch an image with a "best match" platform, which won't work (wrong exec format).  This is the toughest one to fix, since it might require a full recompile of some binary executable files and probably demands building a new image up from multiple layers down.

Troubleshooting

1. Check the image platform
   docker image inspect myimagename:myimagetag |grep -i architecture
2. Check the runtime platform
   docker version |grep -i arch
   or
   kubectl version |grep -i server
3. (Possible) Solutions
1. Pull the image with the platform that matches the target runtime environment
   docker pull --platform linux/amd64 myimagename:myimagetag
   ...then push again to the target registry to overwrite/replace the wrong-platform image.
   See: https://docs.docker.com/build/building/multi-platform/
2. Rebuild the image starting from the correct base image
   Modify Dockerfile and add --platform argument to the FROM line.
   See: https://docs.docker.com/engine/reference/builder/#from

MacOS Display Derangement

Overview

The title is intentional, because MacOS falls short of actually accomplishing display arrangement in some circumstances.

Circumstances

When multiple monitors of certain brands and models are plugged into a MacBook, the part of MacOS that derives a "unique" identifier for each monitor fails to distinguish between them.  The HP E243 is one example, but there may also be other brands/models that reveal this flaw in MacOS' handling of external monitor identification.  Since MacOS cannot tell which specific display was previously arranged in a particular position (relative to the internal display, or other external displays, or each other), the overall arrangement sometimes gets mangled.  The only common way to re-arrange the monitors is by way of the cumbersome "System Preferences -> Displays" dialog in MacOS.

Remedy

After years of searching for a reasonable solution to this issue, and hoping Apple would get it fixed (but has not as of Monterey / 12.6), I finally found, sorta by accident, a command line utility (the best kind IMO) with the source code published on Github, called DisplayPlacer.  DisplayPlacer removes "most" of the annoyance of this Apple MacOS bug.  It still requires a manual step (running a command/script) to force displays back into a captured/desired arrangement, but is WAYYYYYYYY better than using the MacOS "System Preferences -> Displays" UI every time.

Steps

• Install displayplacer (brew tap instructions on the github repo)
• https://github.com/jakehilborn/displayplacer
• Arrange displays using the MacOS System Preferences -> Displays (yup... one last time)
• Run displayplacer list and copy/capture the displayplacer command that is printed to the console, which contains all of the distinct display ids, resolutions, refresh rates, and relative positions.
• Create a shell script containing the captured command.
• Whenever MacOS !@#$% the display arrangement again, run the shell script to pop everything back where it is supposed to go.

Conclusion

This isn't my utility.  The author Jake Hilborn had done us all a favor by making MacOS less annoying while we all continue to wait, maybe indefinitely, for Apple to finally acknowledge this is a problem and fix it themselves.  I would recommend sending Jake a thank you, or maybe find a way to buy him a cup of coffee (or similar).  I only posted this so there will be yet one more way some of you might end up finding Jake's excellent work, and maybe end some of your frustration with the display "derangement" on your MacBook.

Installing webCOrE for SmartThings (Android) - May 2022

Overview

In the wake of several decisions by Samsung and possibly others, the SmartThings eco system is falling apart.  This article sums up several of the answers for installing webCoRE in SmartThings as they are (were) in the spring of 2022.

Alert: If you are trying to do this on an iOS/Apple mobile device, this isn't likely to help you much.  iOS devices aren't really a favorite of people who want to tinker with something like webCoRE anyway, and are generally VERY restrictive, so I'll just wish you luck finding a similar article, if such a thing exists, for your situation.

webCoRE - Why?

Automations in the basic SmartThings app don't allow multiple actions on the same device.  This limitation makes it impossible to do things like using an IOT plug to automate cycling the power off and on for something like an HP multifunction printer that puts itself permanently to sleep under unknown, random circumstances.  webCoRE adds the ability to do more useful automations.

Installation Overview

Instructions found in various places are based on previous versions of the SmartThings IDE (website), and/or previous versions of the SmartThings (Android) mobile app, so the screenshots and instructions often don't match up to current reality.  Even this article will, without a doubt, become obsolete when Samsung moves things around once again.  In the interest of minimizing that "rot" factor, I'm leaving out details for how to do various steps but including enough key words to search the web and find instructions.

The installation process for webCoRE is supposed to be a few simple steps as follows:

1. Add the SmartApps to the SmartThings IDE
• Option 1: Create each of the 4 webCoRE components "from code" and copy/paste the source code into the online editor.
• Option 2: Integrate SmartThings with GitHub, and install each of the 4 webCoRE components from the ady624 GitHub repository.
2. Add a named "instance" of webCoRE within the SmartThings app with access to the devices, sensors, etc. that the instance may control.
• This is where Samsung has made things really obscure and difficult.

Installation "Gotcha's"

SmartThings Classic is Discontinued

• The user interface in SmartThings Classic made it far clearer and easier to install SmartApps, but it is defunct, and the "new" version of the SmartThings app, at least on Android, tries its best to bury and hide the SmartApps functionality.
• If your Android device is pre-"Oreo" (pre Android 8.0), this is probably already a source of frustration, since the "New" SmartThings app won't even install/run.  For instance, a Samsung Galaxy-S 5 makes a pretty good smart remote for IOT stuff, but you'd have to go to some trouble to get Android 8.0 or later installed on it.

Steps to install a SmartApp in the "New" version of SmartThings

• Enable Developer Mode - The option to create a SmartApp won't even show up without doing this first.
• At the moment, this is done by 
• opening the settings (gear icon displayed when the "Menu" page/tab is selected).
• scrolling down to "About SmartThings" and tapping it for 5+ seconds
• scrolling farther down to see, and toggle on "Developer mode"
• Dive Into the Menus to find where they hid the SmartApp option
• At the moment, this is done by
• switching to the Automations page/tab
• clicking the "+" icon and choosing "Add routine"
  
• switching to the "Discover" page/tab
• scrolling all the way to the bottom and selecting the main webCoRE SmartApp (not the dashboard, piston, or storage items)
• Note: These will only appear if they were previously added in the SmartThings IDE

Location

• webCoRE refuses to install unless the Location is properly set in SmartThings.
• Location was apparently managed in the Menu-Settings at some point, but currently, it is "hidden" on the "Favorites" page/tab
• tap the location name (e.g. "Home")
• select "Manage Locations" (gear icon)
• Even if a geolocation is set, it may be too broad for webCoRE to be happy with it, so try reducing the radius, if webCoRE refuses to install.

References

* https://community.webcore.co/t/installation-trouble-webcore-your-location-is-not-correctly-setup/20079

* https://community.smartthings.com/t/faq-did-we-lose-the-ability-to-add-custom-smartapps-after-the-app-update-of-june-2021/227734

* https://community.smartthings.com/t/alternatives-to-smartthings-ecosystem-2021/227572/3

* https://www.youtube.com/watch?v=y_ElUwmmI6Y (overview of install with several outdated screenshots and instructions)

* https://www.reddit.com/r/SmartThings/comments/jwqskf/automations_more_than_one_action_for_the_same/