Wednesday, May 1, 2024

Twilio Trial Accounts are Useless

This post is an attempt to save someone from wasting a lot of time setting up a Twilio trial account in hopes of using it for development or low-volume personal reasons.  With the restrictions on different number types, and the requirements for different registrations or verifications, the trial account, at least for use in the US, is little more than a bait and switch to a paid account.  So, IMO, just save yourself the hassle and leave Twilio for the SMS spammers and big corporations.

There are numerous tutorials, youtube videos, and forum posts that suggest you should be able to register a trial account and use it for development / hobbyist purposes.  Those are apparently mostly obsolete.  

Currently you can:

  • Register a login
  • Create a trial "account" within that login.
  • "Purchase" a phone number with the trial account.
  • Get the credentials (API Key or Auth Token), and
  • Call the API to (try to) send an SMS message.

Then (eventually) find out that the SMS message cannot be delivered, even to a "verified" number, if...

  • The phone number you "purchased" wasn't a toll free number.  Non-toll-free numbers require an A2P 10DLC "campaign" before sending SMS messages is possible. 
    • You can't set up an A2P 10DLC without upgrading from the trial account.
  • You "purchased" a toll-free number but you don't have some kind of business set up already for verification of the toll-free number.
    • That's probably not the case for a hobby/personal/"see how it works" scenario.

I spent several hours trying to navigate Twilio's vague, confusing, incoherent documentation before giving up and opening a ticket with their technical support.  The answer I got from tech support confirmed that there basically isn't a way to use Twilio without upgrading and/or linking it to an established business.  

The "trial" account is effectively a "bait and switch."

Hope this saves someone the hours of time they would waste trying to see how Twilio works.  Hope Twilio realizes that when someone like me, a professional software engineer, has this kind of experience with their (lack of) service, the net result is an ongoing strong recommendations AGAINST using Twilio.

IMO, the tactic of suggesting that something would be free for personal / hobbyist use, even for a limited time, when that is absolutely false, is misleading and dishonest.  If there is ANY alternative, I refuse to engage with such a company. I plan to enthusiastically steer any project I'm working on far away from Twilio

Thursday, February 15, 2024

n8n MongoDB Node - "insert"

This might be the first of several "missing manual" posts for n8n.  The product itself is amazing, but the documentation, and sometimes even the related forum posts are miserably lacking.  This one is about the MongoDB "insert" function.  It should be straightforward and easy, but it does NOT work like you would expect.  If you mouse-hover over the question-mark icons on each input field in the editor form, it helps a little, but still might get you past "trial and error" mode.

What the Docs COULD Say

The value of the "Fields" input must be a comma-separated list of attribute names from the root level of one item in the input data for the node.  For example, if the input data contained this:

[
  {
    "name": "Bob"
    "address": {
      "street": "123 Main St",
      "city": "Lincoln",
      "state": "NE",
      "zip": "68516"
    },
    "age": 42
  },
  {
    "name": "Susan"
    "address": {
      "street": "456 Smith Ave",
      "city": "Springfield",
      "state": "IL",
      "zip": "62704"
    },
    "age": 38
  }
]

...then the MongoDB node insert "Fields" could be...

name, age

 ...and 2 JSON objects (without "address": "value" attributes) would be inserted into the specified collection in MongoDB.

Option: "Use dot notation" (Select/add, then enable)

  • Allows nested attributes to be selected as an included field (excluding peer attributes in the sub-object).
  • This does NOT flatten the structure, but filters which nested fields get included
  • For example, field list could be:

name, age, address.zip

Option: "Date Fields"

  • This is a second comma-separated list field names, but these get converted to MongoDB BSON date values in the inserted object... if you need that for some reason.

 Other notes:

  • Whitespace between fields in the comma separated list of field names will be ignored.
  • If dot notation is specified as a field name without the "Use dot notation" option enabled, the output will most likely include a null attribute like: "somenestedthing.someattribute": null

The Actual "Docs" (Barely)

https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.mongodb/

As of 2/2024, this doesn't tell you much of anything.  If it were the owner's manual for a car, it wouldn't give you much past:  Up to 4 people can fit in it.  It goes.  Hopefully it also stops.

Forums

  • Subject: "Insert JSON data to MongoDB inserting empty values"
    • https://community.n8n.io/t/insert-json-data-to-mongodb-inserting-empty-values/24594
    • This post asks why a single JSON document can't just be inserted into MongoDB as is.
    • The answer runs off in the weeds with some nonsense about splitting things into fields.
  • Subject:  "MongoDB Insert key/value formatting"
    • https://community.n8n.io/t/mongodb-insert-key-value-formatting/4948
    • This post is again from someone confused that the input for a MongoDB insert isn't some sort of reference to both Keys and Values (like a whole JSON document... e.g. $json)
    • The answer is a link to some set of examples that seems completely unrelated.
  • Subject: "MongoDB insert data?"
    • https://community.n8n.io/t/mongodb-insert-data/192
    • This might have been asked before the MongoDB built in node/integration was added.
    • The answer mentions the "new" integration node, but still doesn't explain how to use it.
  • Subject: "How to use the “function node” with a “set node” and a “mongodb insert” in combination?"
    • https://community.n8n.io/t/how-to-use-the-function-node-with-a-set-node-and-a-mongodb-insert-in-combination/21059
    • This SHOULD be included in a search for "fields.split is not a function", but that important bit of text is buried in a screen shot.
    • The answer to the problem most people seem to be having is here if you look carefully, but its one of those things you probably see only after you've figured it out anyway.
  • Subject: "How to insert properly data in mongoDB?"
    • https://community.n8n.io/t/how-to-insert-properly-data-in-mongodb/4052
    • Yet another cry for help, because there is no explanation of how this node works.
    • The replies don't help, and the topic is locked (which is why I'm writing here instead of answering there, btw.)



Saturday, January 27, 2024

IP Masquerading with nftables

Why this is here

I realize I'm sorta late to the game writing this, but then I JUST finished sifting through all the half-a$$'d documentation and examples I could find on the topic and NOT ONE of them offered a clear explanation of what is going on with nftables, and what the parts of a masquerading setup mean.  So here I am, hoping I can do a better job, because I'm gonna forget how this works, and probably come back here later and want me to explain it to future-me.

My other motivation for putting this here is to get feedback in comments in case I don't have some part of this quite right.  If you see something here that is just wrong, please let me know.

The Task

Set up a Linux machine, to forward traffic coming in one network interface, to any of the hosts in the network to which another network interface is attached, on any port.

My reason for doing this was that I had a new Wifi router, that I wanted to set up from my workstation, but my workstation was still connected to the "old" router.  I knew one option would be to create ssh tunnels to forward specific ports to the new router's LAN ip, but that meant opening a terminal and connecting / logging-in with the ssh command every time I wanted to open the router's web interface.  It also meant I'd have to set up one or more port-forwards to reach/test/reconfigure each device as I moved them.  That got tedious fast.

The Plan

I had a Raspberry Pi ZeroW that I had used for some forgotten thing a while back, and it was just sitting there in a box, gathering dust, begging to be made into a mini-nat-bridge/router between my current (old) network and the new one.  So, I searched for stuff like "nat" and "masquerade" and "iptables" because I remember it not being too difficult to get that exact thing working in an older version of Linux.  I even had it working once on a coax network with an old 386 machine running RedHat 2 (or something like that), using ipchains.  (That was so my wife and I could share our dialup connection to Mindspring instead of taking turns... boy those weren't the days).  But, after I installed the latest (Bookworm) Raspberry Pi OS, booted up the pi, connected it to the (old) network, and entered the iptables command, when I saw the dreaded: "command not found", I knew this was a new rabbit hole I had just jumped down.

The New Thing

Shifting my search strategy to things like "what replaced iptables" and "how do you set up masquerading now," I found that NetFilter has moved (or is nearly moved) from iptables to nftables.  I knew from experience that any time there's a subtle name change on something in Linux, right behind that I'll find a drastically different way of doing everything.  nftables didn't disappoint.  Instead of pre-established chains to which you inject your own rules, you now need to create a hierarchical table that contains its own set of chains, and each of those contains its own set of rules.  In a nutshell: 

  • Tables belong to a family that designates what subset of traffic it affects (ipv4, arp, ipv6, etc): https://wiki.nftables.org/wiki-nftables/index.php/Nftables_families
  • Chains are in a table, and belong to a type (filter, route, or nat), and are associated with a set of hooks that map to the part of the packet handling lifecycle related to the type - e.g. nat has hooks like prerouting and forward.
  • Rules are in a Chain, and describe something to be done with a packet.

This wiki page expands upon that "bootstrap view" a little: https://wiki.nftables.org/wiki-nftables/index.php/Quick_reference-nftables_in_10_minutes

How Does NAT Masquerading Work?

Before getting into how to set up nftables for NAT with masquerading, it is helpful to remember what is happening, so:

  • First, some other machine sends packets to the "inbound" network interface that are not destined for the machine that "owns" that interface.  This is typically done with a static route on the machine originating those packets, which is treating that interface as if it is part of a router.
  • If the machine receiving those "not for me" packets has packet-forwarding enabled, it will just send packets along, through whatever "outbound" network interface it would normally choose if it was originating those packets itself.  However, since it doesn't do anything to the fix the return IP address or source port number in those packets, the host that receives them next probably has no idea where to send its response packets, and the round-trip breaks.
  • Masquerading dynamically changes the source IP (to the "outbound" interface's IP) and port (to something it generates and keeps track of) so it can receive and re-route response packets back to the originating source IP and port.
    • Note SNAT is a variation on this same concept, but does not figure out what the "outbound" IP should be.  A SNAT setup depends on a specified, fixed "outbound" IP, which allows it to perform a bit better.

How the nftables Pieces Fit

To eliminate ambiguity between stand-in variables and actual values, this example assumes the following:

  • The machine has two network interfaces (because it needs at least two physically separate connections)
    • "inbound" connection is
      • on network 192.168.5.0/24
      • on an interface named "wlan0"
    • "outbound" connection is
      • on network 192.168.200.0.0/24
      • on an interface named "eth0"
  • The table needs a name, so we'll name it "nat_all_to_network2"
  • Chains can have names, but they'll be named unimaginatively with a shortened form of the hook they reference (e.g. hook prerouting chain will be "pre_rt"
The nft command is used to create tables, chains, and rules.
  • Note: Using the nft command in some circumstances relies on terminating argument/parameter (e.g. -a, -x, etc.) parsing before passing a negative number (e.g. -50).  You will see where that is needed below.
    • In bash at least any dash in the command line, after a double dash ("--") will be parsed as a literal value instead of a parameter - e.g.:  nft -- add ... priority -100
    • https://www.man7.org/linux/man-pages/man1/bash.1.html

Creating the Table

In nftables, a table contains a set of related chains, so that's the first required part.
  • Syntax: nft [command] [object_type] [family] [table_name]
  • Command: nft add table ip nat_all_to_network2
Note: The "ip" family means this rule only affects ipv4 packets.  "inet" would include ipv6 too.
Note: Commands to add chains and rules do not need to repeat the [family] part.

Adding Chains

Getting the table to process the right packets needs two chains which hook into prerouting and postrouting phases of packet handling with a chain type of "nat" (which means it only really examines the first packet of a connection to decide what to do with all of the packets that follow).
Note: Other hook types like "filter" continue evaluating each individual packet.
  • Syntax: nft [command] [object_type] [table_name] { type [chain_type] hook [hook_phase] priority [priority_value] ; }
  • Command (prerouting): nft -- add chain nat_all_to_network2 pre_rt { type nat hook prerouting priority -100 \; }
  • Command (postrouting): nft add chain nat_all_to_network2 post_rt { type nat hook postrouting priority 100 \; }
Note: The semicolon must be escaped so the shell/terminal won't treat it as a multiple-command-separator.
Note: The -100 and 100 priority values map to "special" values named dstnat and srcnat, respectively, which is how they appear when the table is viewed with nft list ruleset
For information overload about this, see: https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks

Adding Rules

The final part of setting up the table is to specify the circumstances when a packet should be evaluated, and what to do with it if it matches various attributes.
The prerouting chain needs a rule that limits the table to evaluate and process only packets coming from the inbound interface
  •  nft add rule nat_all_to_iot_network pre_rt iifname "wlan0"
The postrouting chain needs a rule that tells it to set up masquerading for packets that are being routed to the "outbound" interface.
  • nft add rule nat_all_to_iot_network post_rt oifname "eth0" masquerade

Blocking Reverse Traffic

When the default table in /etc/nftables.conf is loaded, there is a table for the inet family, named "filter", containing chains for input, forward, and output hooks.  To prevent any host on the outbound side from forwarding packets to hosts on the inbound side, add a few rules to the forward chain in this table.

First be sure the return packets from connections forwarded and masqueraded to the outbound side are still forwarded back.
  • nft add rule inet filter forward ct state {established, related} accept
Next drop any packets originating on the outbound side, destined for the inbound side.
  • nft add rule inet filter forward iifname "eth0" oifname "wlan0" drop
Packets that don't match one of those rules default to the chain's "policy accept"

Switching it On

None of this will do anything unless packet forwarding is turned on.
To enable forwarding until the next reboot:
  • sysctl -w net.ipv4.ip_forward=1
To make this permanent across reboots, edit /etc/sysctl.conf and add (probably just uncomment)
  • net.ipv4.ip_forward = 1
To reload the config immediately (to see if everything is set for a reboot), enter commands:
  • sysctl -p  (to reload)
  • sysctl net.ipv4.ip_forward  (to check resulting setting)

Persisting for the Next Boot

After following these directions and testing to be sure packet forwarding is working as expected, the table, chains and rules are just in volatile memory and will vanish upon reboot.  

To persist a table, so it is re-activated every time the system starts, it needs to be added to /etc/nftables.conf.  The format of the tables in the config file is approximately the same as the output of the nft list ruleset command.
  • Note: Special-value priorities displayed in the nft list ruleset command output are shown with their names (e.g. dstnat instead of -100,  and srcnat instead of 100).  nftables.conf must have the numeric values, since it doesn't seem to have a mapping from the names back to their numeric equivalents.
  • Note: Many tutorials found on the internet set up the initial rules locked to a particular network interface using iif and oif.  From the ntf command-line, the interface names seem to be translated into their index values, but in nftables.conf, values like eth0 and wlan0 are actually iifname or oifname, so it is better to set up the rules using those attributes to begin with.
    • See comment on Nov 6, 2023 here: https://superuser.com/questions/1815386/internet-connection-activ-only-after-restart-of-nftables






Friday, December 22, 2023

OpenWRT, dnsmasq, dhcp, static mappings, but No DNS Resolution

The Problem

After poking around WAYYYYY too long to connect these dots, I finally have a reasonable explanation for why dnsmasq, running on on OpenWRT, absolutely refused to resolve an IP address from a hostname that was configured, in DHCP, with an IP for a specific client mac address.  The client machine was online, and reachable by that IP.

TL/DR

dnsmasq only resolves names from DHCP active leases, not from DHCP config.  (Read on if you need to know how to check, how to fix, etc.)

The circumstances were:

  • ip/name configured in DHCP for client machines mac address (as mentioned already).
  • client machine hostname configured to match the name also assigned by DHCP
  • client machine (linux) had been rebooted, after configuring DHCP and hostname
  • command dhclient -r issued repeatedly on the client machine
  • interface, domain, etc. set to "lan"
  • ipv6 disabled (just in case it was making things more complicated).
  • ping and nslookup from a different machine received NXDOMAIN response from dnsmasq/openwrt

Extra Circumstances (that didn't seem relevant, but were):

  • client machine had unexpired, cached DHCP lease info in /var/lib/dhcp/dhclient.eth0.leases
  • openwrt dhcp did NOT have an active lease registered in /tmp/dhcp.leases
  • dnsmasq had been restarted, the entire openwrt router had been restarted, and several updates had been "Saved and Applied" via LuCI.  This probably meant that any DHCP in-memory name:ip mappings were gone.

What fixed it?

On the client/host machine that can't be resolved/found by name...
  1. Delete /var/lib/dhcp/dhclient*.leases
  2. Reboot (because dhclient -r eth0 didn't help).

Verification

  1. Check that ping and nslookup now work as expected from other clients for the previously (let's hope) unresolvable hostname.
  2. Check that openwrt /tmp/dhcp.leases DOES now include a line for the recently, forcibly-renewed DHCP lease.

Explanation:

openwrt runs dnsmasq with a configuration that references the /tmp/dhcp.leases file.  
  • The setting is found on the Network - DHCP and DNS page in LuCI, on the Resolv and Hosts Files "tab" in the Leasefile field.
  • The setting is found in the /etc/config/dhcp file in the config dnsmasq section as option leasefile '/tmp/dhcp.leases'
    • See: https://openwrt.org/docs/guide-user/base-system/dhcp
    • This says: "leasefile stores leases in a file so they can be picked up again if dnsmasq is restarted."
    


Monday, December 18, 2023

Gogs: Internal Error from "git push"

This might have been fixed in Gogs, and maybe it just got me because I'm just running an older version, but I think it's worth a quick post here anyway...

Problem

Pushing a git repository to a remote hosted in Gogs... with ssh config, etc. all as it should be... results in:

me@myworkstation#:~/mylocalgitclonedir $ git push

Gogs: Internal error

fatal: Could not read from remote repository.


Please make sure you have the correct access rights

and the repository exists.

Cause

It turned out to be root:root ownership of the xorm.log file on the Gogs server.

In my case, running in a docker container, that was /app/gogs/log/xorm.log

This might be related: https://github.com/gogs/gogs/issues/4990

Don't know how this got changed in my particular setup... yet.  Maybe a backup or log rotation hiccup.

So, the cause was not: (including some of this for searchability of this article)

  • repository write permissions for user
  • SSH config on server or client
    • i.e. not an ssh client known_hosts error
    • i.e. not related to private key file permissions on the client
    • i.e. not using the wrong private key for the registered public key in gogs
    • checked with command:  ssh -v -T git@mygogshost

Solution

Open a terminal on the gogs host, and...

sudo chown git:git /app/gogs/log/xorm.log

After that git push worked fine again.

References

  • https://github.com/gogs/gogs/issues/4990

Tuesday, December 5, 2023

HomeAdvisor.com Password Reset

This info is current as of Dec 5, 2023, so if you're reading this later, and that was ages ago, it's not likely to be correct still.  So, with that said...

I just had the "wonderful" experience of trying to get logged in to HomeAdvisor.com on a mobile browser after initiating an account from a desktop web browser.

Their "clever" scheme to use a temporary login link, sent to an email address, instead of a password, makes it impossible to CHANGE your password once you are logged in, because you might have NEVER SET a password.  The "Forgot Password" link just sends you the non-password login link again... which is just another trip around their endless loop.

Called customer support and described the issue, and they just hung up on me, which might be an indication of whether you'd even want to work with HomeAdvisor/Angi in the first place.

The only way I could get out of the "catch 22" they created on the HomeAdvisor.com site, was to guess that maybe all of the the accounts are wired into the same place as Angi.com.  So, if you go to Angi.com's login page:  https://www.angi.com/login , you'll find a "Forgot Password? link that still goes to another page, which asks for your email address (i.e. login id), and actually sends you a link to reset your password on angi.com.  Magically, that ALSO sets your password on HomeAdvisor.com.

Since the idiots in customer support can't be bothered to help anyone figure out the !@#$% mess they've made of the logins between HomeAdvisor.com and Angi.com, I hope this helps, at least until they fix it, or (more likely) make it into an even bigger mess.

Friday, October 13, 2023

Legacy NVIDIA Driver for Ubuntu 20.04+

Summary

Either because an upgrade wrecked a working Linux install, or because the default drivers aren't performing very well, it may be necessary to install NVIDIA drivers for very old, "legacy" hardware on Ubuntu, or a similar Linux distribution.  This post is meant to help get that done without burning as much time searching for the right information and putting all the pieces together.

Symptoms

  • After an update / upgrade, Linux stalls mid-boot with a black screen and a blinking text cursor in the upper left corner.
  • Linux installed on a machine with older / legacy NVIDIA hardware is running too slow to be usable, and the graphics driver is suspected as the cause.

Elements of the Problem

  • Recent versions of Ubuntu install a kernel version that is no longer compatible with older NVIDIA drivers.
  • The default, generic driver doesn't perform very well (or at all) with older NVIDIA graphics card hardware.

Getting to a Terminal

When/if the X-Window / GUI Startup failed and stopped the bootup, it may be necessary to switch to another tty/terminal.  There are typically several of them started with Linux, and they can be accessed using CTRL+ALT+F2 (or F3, F4, ...).  From the blank / black screen with the blinking cursor, use that key combo to switch over and directly log-in to a terminal / tty, where the commands below can be executed.

Diagnostic Steps

  • Verify old NVIDIA hardware
    • Command:   sudo lspci | grep VGA
      • Should display information about the NVIDIA hardware (e.g. GeForce 9600M GT)
  • Verify old, no longer working, NVIDIA driver version is installed - (e.g. 340.108)
    • Command: dkms status
    • Command: apt list --installed | grep nvidia
      • Should show the NVIDIA driver package version (e.g. nvidia-340 ... 340.108)
  • Verify NVIDIA driver is malfunctioning
    • Command: nvidia-smi
      • Should report a failure to communicate with graphics hardware.
  • Check NVIDIA settings
    • Command: nvidia-settings
      • May report config errors, etc. and/or other useful info.

Elements of the Solution

  • Add the PPA repository from user kelebek333 which contains a version of the NVIDIA drivers that has been patched to work with newer kernel versions.
    • See: https://launchpad.net/~kelebek333/+archive/ubuntu/nvidia-legacy
    • Command: sudo add-apt-repository ppa:kelebek333/nvidia-legacy
  • Run command: sudo apt-get update
    • This should report that one or more packages (which include patched NVIDIA driver) are upgradable.
      • Verify w/ command: apt list --upgradable
  • Run command: sudo apt upgrade nvidia-340
    • This may report that there is a public key missing.  If so, run the following command replacing {key-value} with the key that is reported in the error message from apt upgrade...
      • Command: sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys {key-value}
  • Reboot - If this resolved the NVIDIA driver issues, the X Window system should start normally again.