I have a lump on my left nostril cartilage that pops out and I can push it back in when I hold it. It’s painless and skin-colored but feels hard when touching and affects the shape of my nostril and breathing.
It sounds like you are talking about your lower lateral cartilage. Take a look at some images on it and see if this correlates. There can be a lot of variability in its shape between, and within, individuals.