How to deny an ip address from connecting to my system?

You can use ufw as well, and IMO, it's much simpler.

To deny an IP address from any connections use:

sudo ufw deny from <ip_address> to any  

Or to deny access to a specific port:

sudo ufw deny from <ip_address> to any port <some_port>  

or you can specify protocol:

sudo ufw deny proto tcp from <ip_address> to any port <some_port>  

(replacing tcp with either tcp or udp depending on what protocol you want)

Before we go any further, lets make note of a couple things.

1. The majority of the Internet does not have static IP addresses. Therefore, proxies, Tor, etc. can be used to try and bypass single-IP blocks.
2. It is usually safer to do the inverse, to block all traffic on a port, and accept only certain addresses. This is not feasible in most web-facing situations, but it is still a generally-accepted security practice.
3. We do not do any filtering in here on the Internet in general - we do not take into account point #1 with the below answer, and you must consider the likelihood that if you block one IP, they'll find a way around it and get access to a different IP, and continue attacking.

Before anyone comments that ufw is less complicated than iptables, I used iptables here because the OP answered this question using iptables commands, which led me to assume they wanted an answer that had iptables commands and not ufw commands/

A good iptables firewall configuration will deny everything except the bare minimum requirements that you need on your system(s) in terms of "acceptable connections", and should only accept connections that have been explicitly allowed in the rules.

To do this, we have to configure iptables AND ip6tables to block traffic by default, except traffic you've initiated from inside your computer (such as web traffic from Firefox, etc., if you're on a system, or a wget stream to download software, or to allow apt to update the packages). We also have to allow you to be able to add acceptable rules for traffic you WANT to have reaching your system.

In order for these rules to be persistent so they work all the time on every reboot, you should install the iptables-persistent package, which will allow you to save these rules and then automatically load them at system boot. Run sudo apt-get install iptables-persistent first, then add your rules. We'll make sure to update the ruleset that is loaded at boot after we've created a rule set.

Assuming we've started with a blank iptables and ip6tables setup (which is the default in Ubuntu), you'll need to add a bunch of rules. I'll start with iptables, then we'll work with ip6tables.

iptables

The following is a basic ruleset that will restrict all data and packets attempting to reach your system, regardless of what connection you're on:

Accept localhost data
iptables -A INPUT -i lo -j ACCEPT - accept all traffic on LocalHost, which is traffic that is local to within the computer only. This is necessary for your computer to function correctly.

Accept data related to outbound connections that your system has initiated
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT - accept all traffic that is related to connections going in an outbound direction, including responses from remote servers (such as the apt mirrors, or any website you're visiting).

ICMP rules
Note that I do not filter out ICMP packets, as those're generally OK to accept, unless you're paranoid. If you really want to block ICMP, then you need to use these rules to accept some critical packets:

Accept "Time Exceeded" ICMP
iptables -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT - accept "Time Exceeded" packets which is necessary for certain time-restricted connection setups.

Accept "Destination Unreachable" ICMP
iptables -A INPUT -p icmp -m icmp --icmp-type 3/4 -j ACCEPT - accept "Destination Unreachable" ICMP responses from remote servers. This allows the system to operate correctly with remote sites which you can't reach.

Accept PING requests/responses ("Echo" ICMP)
iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT - accept "Echo" (aka "Ping") requests. This is good to keep around because your system's connection to certain sites and/or services may require a PING for a keep-alive request.

Create an iptables table to hold acceptable traffic ports/rules
iptables -N acceptable_traffic
iptables -A INPUT -j acceptable_traffic

By doing this, you have a table where you can put rules for what traffic you wish to accept on your system from external connections. If you have an SSH server, you'd do iptables -A acceptable_traffic -p tcp --dport 22 -j ACCEPT (or replace 22 with the SSH server port). You'd add other acceptable traffic rules here too.

Reject all traffic for which there are no rules set
There's two options for this.

iptables -A INPUT -j REJECT --reject-with icmp-host-unreachable - This is the first option, and what I consider a more sane way of blocking traffic, when an individual attempts to reach the server, they'll receive a "Host Unreachable" ICMP packet which then terminates the connection attempt.

iptables -A INPUT -j DROP - This is the second option. This will automatically drop all incoming packets that're sent to your system which don't match an ACCEPT rule. This means the packets are completely ignored. The connection attempts from the remote location will just time out on their own with no response from the system being connected to.

ip6tables

Like iptables for IPv4, you have to configure ip6tables for IPv6. The rules are similar, but different. *If you do not have IPv6 on your system, you may not have to worry about this, however having the rules in the system won't hurt.

Accept localhost data
ip6tables -A INPUT -i lo -j ACCEPT - accept all traffic on LocalHost, which is traffic that is local to within the computer only. This is necessary for your computer to function correctly.

Accept data related to outbound connections that your system has initiated
ip6tables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT - accept all traffic that is related to connections going in an outbound direction, including responses from remote servers (such as the apt mirrors, or any website you're visiting).

ICMPv6 rules
ip6tables -A INPUT -p icmpv6 -j ACCEPT - I do not block IPv6 ICMP packets, because unlike IPv4, there's a billion more IPv6 ICMP packets that you should accept (i.e. all of them). To that end, you can do this to accept all of them.

Create an ip6tables table to hold acceptable traffic ports/rules
ip6tables -N acceptable_traffic
ip6tables -A INPUT -j acceptable_traffic

By doing this, you have a table where you can put rules for what traffic you wish to accept on your system from external connections. If you have an SSH server, you'd do ip6tables -A acceptable_traffic -p tcp --dport 22 -j ACCEPT (or replace 22 with the SSH server port). You'd add other acceptable traffic rules here too.

Reject all traffic for which there are no rules set
There's two options for this.

ip6tables -A INPUT -j REJECT --reject-with icmp6-addr-unreachable - This is the first option, and what I consider a more sane way of blocking traffic, when an individual attempts to reach the server, they'll receive an "Address Unreachable" ICMPv6 packet which then terminates the connection attempt.

ip6tables -A INPUT -j DROP - This is the second option. This will automatically drop all incoming packets that're sent to your system which don't match an ACCEPT rule. This means the packets are completely ignored. The connection attempts from the remote location will just time out on their own with no response from the system being connected to.

Save all the rules

Since I told you earlier to install the iptables-persistent package, you'll need to actually save the rules so they can be loaded later.

NOTE: This set of commands actually drops you into a superuser shell. YOU SHOULD NOT DO ANYTHING ELSE IN THIS SHELL EXCEPT THESE COMMANDS I STATE HERE!

Execute the following commands:

sudo -s
iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6
exit

This will save the rules so they can be loaded at every reboot. IF YOU EDIT THE RULES LATER, YOU WILL NEED TO RERUN THESE COMMANDS TO SAVE THE RULESETS.

What if I actually want a list of bad IPs to block?

We can still do this! You may want a list of blocks for 'bad' traffic. So, here's my solution for this, and like before I'll split it into two sections, iptables and ip6tables.

iptables

Create a table for prohibited_traffic

We need to now create a prohibited_traffic table. You'll add 'bad IPs' here. We then need to insert the prohibited_traffic rule into the INPUT table. (In the example below, I use 2 to indicate the rule number location for where we insert this rule, which puts it right after the rule that reads ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED:

iptables -N prohibited_traffic
iptables -I INPUT 2 -j prohibited_traffic

Add rules to block certain IPs

Now we add IPs to block to this prohibited_traffic table. Here's the template:

iptables -A prohibited_traffic -s SourceIPAddress -j DROP

Replace 'SourceIPAddress' with the IP address you want to block. If you have a range of IPs to block you will need to take the 'range' and determine the CIDR range that contains all the IP addresses, and then use the format IPV4ADDRESS/cidr to block that CIDR range that includes the IPs. Range blocks can't be done without CIDR or adding a LOT of IPs one by one.

ip6tables

This is almost identical to iptables

Create a table for prohibited_traffic

We need to now create a prohibited_traffic table. You'll add 'bad IPs' here. We then need to insert the prohibited_traffic rule into the INPUT table. (In the example below, I use 2 to indicate the rule number location for where we insert this rule, which puts it right after the rule that reads ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED:

ip6tables -N prohibited_traffic
ip6tables -I INPUT 2 -j prohibited_traffic

Add rules to block certain IPs

Now we add IPs to block to this prohibited_traffic table. Here's the template:

ip6tables -A prohibited_traffic -s SourceIPAddress -j DROP

Replace 'SourceIPAddress' with the IP address you want to block. If you have a range of IPs to block you will need to take the 'range' and determine the CIDR range that contains all the IP addresses, and then use the format IPV6ADDRESS/cidr to block that CIDR range that includes the IPs. Range blocks can't be done without CIDR or adding a LOT of IPs one by one.

One of the options is using iptables that will activate a real firewall to your system; this is probably the most safe way (see the other answers).

This answer is useful if the problem is that you see repeated attempt to enter in your system via the sshd daemon (SSH server service).

Notice: as per 14.04, denyhost support has been dropped.

The new solution is to use fail2ban. It's more powerful and as easy as it gets to install with the default configuration. See http://bodhizazen.com/Tutorials/SSH_security

old answer

You haven't explicitly stated which kind of problem you see, but if the problem is that someone is trying to enter your system via brute-force ssh attacks (hardly dangerous if your system is well configured, but still annoying), I suggest you have a look at DenyHosts.

You can install it from the repository with apt-get install denyhosts and it just works out of the box, blocking automatically for a week any host which attempts to login in your machine and fails three times.

iptables -A INPUT -s IP-ADDRESS -j DROP

Replace IP-ADDRESS with your actual IP address. For example, if you wish to block an ip address 192.168.0.100 for whatever reason then type the command as follows:

iptables -A INPUT -s 192.168.0.100 -j DROP

If you just want to block access to one port from an ip 192.168.0.100 to port 80 then type command:

iptables -A INPUT -s 192.168.0.100 -p tcp --destination-port 80 -j DROP