Trouble Configuring Subdomain on Digital Ocean
Your initial configuration that just had an A record for app.example.com pointing to the new droplet IP was correct. The additional NS records you added were unnecessary.
The connection refused message you got with the initial configuration likely points to a non-DNS related config issue on your droplet unless you typo'd the IP address in the A record. Check firewalls. Check that the web server is running and on the port you expect.