How do I remove non-uniform spacing between my table elements?

I am trying to make a paint website and for showing the colour palette, I use an HTML table of buttons that looks something like the following:
enter image description here

As you can see, the first three elements of both the rows are not uniformly spaced as the other elements in that row. To see why this problem must be occurring, I also used inspect but I couldn’t figure it out from there too because that gap wasn’t being shown in the margin, border or padding of the buttons.

enter image description here

I can’t understand this gap mainly because I have used uniform spacing and sizes for all the elements(You can see in the code snippet below). How do I correct this?

html {
    min-width: 1500px;
    position: relative;
    }

#toolset {
    width     : 100px;
    height    : 340px;
    position  : absolute;
    left      : 0px;
    top       : 50px; 
    background: #35d128;
}

#paint {
    position  : absolute;
    left      : 130px;
    top       : 50px;
}

#colorset {
    position  : absolute;
    left      : 0px;
    top       : 450px;
    width     : 300px;
}

#colorset td {
    width     : 18px;
    height    : 20px;
}
 
#title {
    position  : absolute;
    left      : 500px;
}

#penciltool {
    background: #358128; 
    color     : #f3f3f3;
    width     : 80px;
    height    : 25px;
    border    : 1px solid #33842a;
    -webkit-border-radius: 0 15px 15px 0;
    -moz-border-radius: 0 15px 15px 0;
    box-shadow: rgba(0, 0, 0, .75) 0 2px 6px;
}

#linetool {
    background: #358128; 
    color     : #f3f3f3;
    width     : 80px;
    height    : 25px;
    border    : 1px solid #33842a;
    -webkit-border-radius: 0 15px 15px 0;
    -moz-border-radius: 0 15px 15px 0;
    box-shadow: rgba(0, 0, 0, .75) 0 2px 6px;
}

#rectangletool {
    background: #358128; 
    color     : #f3f3f3;
    width     : 80px;
    height    : 25px;
    border    : 1px solid #33842a;
    -webkit-border-radius: 0 15px 15px 0;
    -moz-border-radius: 0 15px 15px 0;
    box-shadow: rgba(0, 0, 0, .75) 0 2px 6px;
}

#circletool {
    background: #358128; 
    color     : #f3f3f3;
    width     : 80px;
    height    : 25px;
    border    : 1px solid #33842a;
    -webkit-border-radius: 0 15px 15px 0;
    -moz-border-radius: 0 15px 15px 0;
    box-shadow: rgba(0, 0, 0, .75) 0 2px 6px;
}

#erasertool {
    background: #358128; 
    color     : #f3f3f3;
    width     : 80px;
    height    : 25px;
    border    : 1px solid #33842a;
    -webkit-border-radius: 0 15px 15px 0;
    -moz-border-radius: 0 15px 15px 0;
    box-shadow: rgba(0, 0, 0, .75) 0 2px 6px;
}

#resettool {
    background: #358128; 
    color     : #f3f3f3;
    width     : 80px;
    height    : 25px;
    border    : 1px solid #33842a;
    -webkit-border-radius: 0 15px 15px 0;
    -moz-border-radius: 0 15px 15px 0;
    box-shadow: rgba(0, 0, 0, .75) 0 2px 6px;
}                
<fieldset id="colorset" style="margin-top: 1.8%;">

        <table>
            <tr>
                <td><button style="height: 18px; width: 80px;" onclick="fill()">Fill</button>
                <td><button style="background-color: #000000; height: 15px; width: 15px;" onclick="color('#000000')"></button>
                <td><button style="background-color: #B0171F; height: 15px; width: 15px;" onclick="color('#B0171F')"></button>
                <td><button style="background-color: #DA70D6; height: 15px; width: 15px;" onclick="color('#DA70D6')"></button>
                <td><button style="background-color: #8A2BE2; height: 15px; width: 15px;" onclick="color('#8A2BE2')"></button>
                <td><button style="background-color: #0000FF; height: 15px; width: 15px;" onclick="color('#0000FF')"></button>
                <td><button style="background-color: #4876FF; height: 15px; width: 15px;" onclick="color('#4876FF')"></button>
                <td><button style="background-color: #CAE1FF; height: 15px; width: 15px;" onclick="color('#CAE1FF')"></button>
                <td><button style="background-color: #6E7B8B; height: 15px; width: 15px;" onclick="color('#6E7B8B')"></button>
                <td><button style="background-color: #00C78C; height: 15px; width: 15px;" onclick="color('#00C78C')"></button>
                <td><button style="background-color: #00FA9A; height: 15px; width: 15px;" onclick="color('#00FA9A')"></button>
                <td><button style="background-color: #00FF7F; height: 15px; width: 15px;" onclick="color('#00FF7F')"></button>
                <td><button style="background-color: #00C957; height: 15px; width: 15px;" onclick="color('#00C957')"></button>
                <td><button style="background-color: #FFFF00; height: 15px; width: 15px;" onclick="color('#FFFF00')"></button>
                <td><button style="background-color: #CDCD00; height: 15px; width: 15px;" onclick="color('#CDCD00')"></button>
                <td><button style="background-color: #FFF68F; height: 15px; width: 15px;" onclick="color('#FFF68F')"></button>
                <td><button style="background-color: #FFFACD; height: 15px; width: 15px;" onclick="color('#FFFACD')"></button>
                <td><button style="background-color: #FFEC8B; height: 15px; width: 15px;" onclick="color('#FFEC8B')"></button>
                <td><button style="background-color: #FFD700; height: 15px; width: 15px;" onclick="color('#FFD700')"></button>
                <td><button style="background-color: #F5DEB3; height: 15px; width: 15px;" onclick="color('#F5DEB3')"></button>
                <td><button style="background-color: #FFE4B5; height: 15px; width: 15px;" onclick="color('#FFE4B5')"></button>
                <td><button style="background-color: #EECFA1; height: 15px; width: 15px;" onclick="color('#EECFA1')"></button>
                <td><button style="background-color: #FF9912; height: 15px; width: 15px;" onclick="color('#FF9912')"></button>
                <td><button style="background-color: #8E388E; height: 15px; width: 15px;" onclick="color('#8E388E')"></button>
                <td><button style="background-color: #7171C6; height: 15px; width: 15px;" onclick="color('#7171C6')"></button>
                <td><button style="background-color: #7D9EC0; height: 15px; width: 15px;" onclick="color('#7D9EC0')"></button>
                <td><button style="background-color: #388E8E; height: 15px; width: 15px;" onclick="color('#388E8E')"></button>
                
            </tr>
            
            <tr>
                <td><button style="height: 18px; width: 80px" onclick="outline()">Outline</button>
                <td><button style="background-color: #71C671; height: 15px; width: 15px;" onclick="color('#71C671')"></button>
                <td><button style="background-color: #8E8E38; height: 15px; width: 15px;" onclick="color('#8E8E38')"></button>
                <td><button style="background-color: #C5C1AA; height: 15px; width: 15px;" onclick="color('#C5C1AA')"></button>
                <td><button style="background-color: #C67171; height: 15px; width: 15px;" onclick="color('#C67171')"></button>
                <td><button style="background-color: #555555; height: 15px; width: 15px;" onclick="color('#555555')"></button>
                <td><button style="background-color: #848484; height: 15px; width: 15px;" onclick="color('#848484')"></button>
                <td><button style="background-color: #F4F4F4; height: 15px; width: 15px;" onclick="color('#F4F4F4')"></button>
                <td><button style="background-color: #EE0000; height: 15px; width: 15px;" onclick="color('#EE0000')"></button>
                <td><button style="background-color: #FF4040; height: 15px; width: 15px;" onclick="color('#FF4040')"></button>
                <td><button style="background-color: #EE6363; height: 15px; width: 15px;" onclick="color('#EE6363')"></button>
                <td><button style="background-color: #FFC1C1; height: 15px; width: 15px;" onclick="color('#FFC1C1')"></button>
                <td><button style="background-color: #FF7256; height: 15px; width: 15px;" onclick="color('#FF7256')"></button>
                <td><button style="background-color: #FF4500; height: 15px; width: 15px;" onclick="color('#FF4500')"></button>
                <td><button style="background-color: #F4A460; height: 15px; width: 15px;" onclick="color('#F4A460')"></button>
                <td><button style="background-color: #FF8000; height: 15px; width: 15px;" onclick="color('FF8000')"></button>
                <td><button style="background-color: #FFD700; height: 15px; width: 15px;" onclick="color('#FFD700')"></button>
                <td><button style="background-color: #8B864E; height: 15px; width: 15px;" onclick="color('#8B864E')"></button>
                <td><button style="background-color: #9ACD32; height: 15px; width: 15px;" onclick="color('#9ACD32')"></button>
                <td><button style="background-color: #66CD00; height: 15px; width: 15px;" onclick="color('#66CD00')"></button>
                <td><button style="background-color: #BDFCC9; height: 15px; width: 15px;" onclick="color('#BDFCC9')"></button>
                <td><button style="background-color: #76EEC6; height: 15px; width: 15px;" onclick="color('#76EEC6')"></button>
                <td><button style="background-color: #40E0D0; height: 15px; width: 15px;" onclick="color('#40E0D0')"></button>
                <td><button style="background-color: #9B30FF; height: 15px; width: 15px;" onclick="color('#9B30FF')"></button>
                <td><button style="background-color: #EE82EE; height: 15px; width: 15px;" onclick="color('#EE82EE')"></button>
                <td><button style="background-color: #FFC0CB; height: 15px; width: 15px;" onclick="color('#FFC0CB')"></button>
                <td><button style="background-color: #7CFC00; height: 15px; width: 15px;" onclick="color('#7CFC00')"></button>
            </tr>
            <tr>
                <td><label>Line Width</label></td>
                <td><button id="pixel_plus" type="button" onclick="add_pixel()" style="width: 25px;">+</button></td>
                <td><button id="pixel_minus" type="button" onclick="reduce_pixel()" style="width: 25px;">-</button></td>
                <td><button id="undo" type="button" onclick="undo_pixel()"style="width: 75px;">Undo</button></td>

            </tr>
            </table>
            <br>
            <label>Tag</label><input id="fname" type="text"><button id="save" onclick="temp()">Add</button>
            <br>
            <br>
            </fieldset>